Transmitting apparatus, receiving apparatus, and transmission system

ABSTRACT

It is an object to realize a correcting process for correcting a defective image in a region of interest (ROI) that is a partial region segmented from a captured image. A transmitting apparatus includes a controlling section that controls the holding of defect correcting information for use in correcting a defect in an image included in a ROI and a transmitting section that sends out image data of the image included in the ROI as payload data and sends out ROI information as embedded data.

TECHNICAL FIELD

The present disclosure relates to a transmitting apparatus, a receivingapparatus, and a transmission system.

BACKGROUND ART

In recent years, there have been growing applications in which largeamounts of data are transmitted in bulk. Such applications tend to poselarge loads on the transmission system, possibly causing thetransmission system to go down in worst-case scenarios and fail toperform data transmission.

To avoid transmission system shutdowns, it has been known in the art tospecify an object as an imaging target and transmit only a partial imageof the specified object that has been segmented, rather thantransmitting an entire captured image (see, for example, PTL 1 throughPTL 4). Furthermore, PTL 5 and PTL 6 disclose a process for correcting adefective pixel in image capturing elements.

CITATION LIST Patent Literature [PTL 1]

-   Japanese Patent Laid-open No. 2016-201756

[PTL 2]

-   Japanese Patent Laid-open No. 2014-39219

[PTL 3]

-   Japanese Patent Laid-open No. 2013-164834

[PTL 4]

-   Japanese Patent Laid-open No. 2012-209831

[PTL 5]

-   Japanese Patent Laid-open No. 2003-163842

[PTL 6]

-   Japanese Patent Laid-open No. 2012-100166

SUMMARY Technical Problem

However, nothing has been examined about a correcting process forcorrecting a defective image in a case where a partial region ofinterest (ROI) segmented from a captured image is transmitted.

It is an object of the present disclosure to realize a correctingprocess for correcting a defective image in a region of interest (ROI)that is a partial region segmented from a captured image.

Solution to Problem

A transmitting apparatus according to an aspect of the presentdisclosure includes a controlling section that controls holding ofdefect correcting information for use in correcting a defect in an imageincluded in a ROI (Region Of Interest) and a transmitting section thatsends out image data of the image included in the ROI as payload dataand sends out ROI information as embedded data.

A receiving apparatus according to an aspect of the present disclosureincludes a receiving section that receives a transmission signalincluding image data of an image included in a ROI (Region Of Interest)in payload data and including ROI information in embedded data, acontrolling section that controls extraction of defect correctinginformation for use in correcting a defect in the image data of theimage included in the ROI from the transmission signal received by thereceiving section, and a processing section that processes a correctionof the defect in the image of the ROI on the basis of the defectcorrecting information extracted by the controlling section.

A transmission system according to an aspect of the present disclosureincludes a transmitting apparatus having a controlling section thatcontrols holding of defect correcting information for use in correctinga defect in an image included in a ROI (Region Of Interest) and atransmitting section that sends out image data of the image included inthe ROI as payload data and sends out ROI information as embedded data,and a receiving apparatus having a receiving section that receivestransmission signal including the image data of the image included inthe ROI in the payload data and including the ROI information in theembedded data, a controlling section that controls extraction of defectcorrecting information for use in correcting the defect in the imagedata of the image included in the ROI from the transmission signalreceived by the receiving section, and a processing section thatprocesses a correction of the defect in the image of the ROI on thebasis of the defect correcting information extracted by the controllingsection.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a general configurational example of avideo transmission system.

FIG. 2 is a diagram illustrating a general configurational example of avideo transmitting apparatus illustrated in FIG. 1.

FIG. 3 is a diagram illustrating an example of a procedure forgenerating transmission data when two ROIs are included in a capturedimage.

FIG. 4 is a diagram illustrating a configurational example of a packetheader.

FIG. 5 is a diagram illustrating a configurational example oftransmission data.

FIG. 6 is a diagram illustrating a configurational example oftransmission data.

FIG. 7 is a diagram illustrating a configurational example of thepayload data of a long packet.

FIG. 8 is a diagram illustrating a general configurational example of avideo receiving apparatus illustrated in FIG. 1.

FIG. 9 is a diagram illustrating an example of a procedure forgenerating two ROI images included in a captured image when two imagesare included in transmission data.

FIG. 10 is a diagram schematically illustrating regions where objectsspecified in a captured image are placed.

FIG. 11 is a diagram illustrating an example of ROIs established withrespect to the specified objects.

FIG. 12 is a diagram illustrating a configurational example oftransmission data where the positional information of ROI images isincluded in the payload data of a long packet.

FIG. 13 is a diagram schematically illustrating a correcting process forcorrecting a defective image according to the present disclosure.

FIG. 14 is a diagram schematically illustrating a method of calculatingthe position of a defective pixel in a region of interest on the basisof coordinate information of the defective pixel acquired in a deliveryinspection or the like according to a first embodiment.

FIG. 15 is a block diagram illustrating a general makeup of atransmitting apparatus, a receiving apparatus, and a transmission systemaccording to the first embodiment.

FIG. 16 is a flowchart illustrating an example of sequence of a processfor acquiring coordinates of a defective pixel in an initializingprocess of the transmitting apparatus, the receiving apparatus, and thetransmission system according to the first embodiment.

FIG. 17 is a flowchart illustrating an example of a sequence of acalculating process for the coordinate conversion of a defective pixelupon normal operation of the transmitting apparatus, the receivingapparatus, and the transmission system according to the firstembodiment.

FIG. 18 is a flowchart illustrating an example of a sequence of acorrecting process for correcting a defective image upon normaloperation of the transmitting apparatus, the receiving apparatus, andthe transmission system according to the first embodiment.

FIG. 19 is a diagram illustrating an example of a timing chart of thecorrecting process for correcting a defective image in the transmittingapparatus, the receiving apparatus, and the transmission systemaccording to the first embodiment.

FIG. 20 is a block diagram illustrating a general makeup of atransmitting apparatus, a receiving apparatus, and a transmission systemaccording to Modification 1 of the first embodiment.

FIG. 21 is a block diagram illustrating a general makeup of atransmitting apparatus, a receiving apparatus, and a transmission systemaccording to Modification 2 of the first embodiment.

FIG. 22 is a block diagram illustrating a general makeup of atransmitting apparatus, a receiving apparatus, and a transmission systemaccording to Modification 3 of the first embodiment.

FIG. 23 is a diagram schematically illustrating a method of calculatingthe position of a defective pixel in a region of interest on the basisof coordinate information of the defective pixel acquired in a deliveryinspection or the like according to a second embodiment.

DESCRIPTION OF EMBODIMENTS

Modes for carrying out the present disclosure will be described indetail hereinbelow with reference to the drawings. The description givenbelow applies to specific examples of the present disclosure, and thepresent disclosure is not limited to the aspects illustrated below.

The modes for carrying out the technology according to the presentdisclosure (hereinafter referred to as “embodiments”) will be describedhereinbelow in the following order:

1. Technology 1 that is presupposed for the present disclosure(technology for transmitting a partial region (rectangular in shape) ofinterest (ROI) segmented from a captured image)

2. Technology 2 that is presupposed for the present disclosure(technology for transmitting a partial region (non-rectangular in shape)of interest (ROI) segmented from a captured image)

3. Principles of a correcting process for correcting a defective pixelin an embodiment of the present disclosure

4. A transmitting apparatus, a receiving apparatus, and a transmissionsystem according to a first embodiment of the present disclosure

5. A transmitting apparatus, a receiving apparatus, and a transmissionsystem according to a modification of the first embodiment

6. Principles of a correcting process for correcting a defective pixelin an second embodiment of the present disclosure

1. Technology 1 that is Presupposed for the Present Disclosure[Configuration]

In recent years, portable devices such as smartphones and camera deviceshave been handling progressively larger quantities of image data, andare required to speed up and consume less electric power for datatransmission within themselves or between different devices. In order tomeet such requirements, standardization is under way for high-speedinterface standards such as C-PHY standards and D-PHY standardsestablished by MIPI Alliance as connection interfaces for potable deicesand camera devices. The C-PHY standards and D-PHY standards areinterface standards for physical layers (PHY) of communicationprotocols. In addition, DSI for the displays of portable devices and CSIfor camera devices are present as higher protocol layers than the C-PHYstandards and D-PHY standards.

A video transmission system 1 according to the technology that ispresupposed for the present disclosure includes a system fortransmitting and receiving signals according to various standards, andcan transmit and receive signals according to the MIPI CSI-2 standards,the MIPI CSI-3 standards, or the MIPI DSI standards, for example. FIG. 1illustrates a general configuration of the video transmission system 1according to the technology that is presupposed for the presentdisclosure. The video transmission system 1 is applied to thetransmission of data signals, clock signals, and control signals, andincludes a video transmitting apparatus 100 and a video receivingapparatus 200. The video transmission system 1 includes a data lane DLfor transmitting data signals representing image data etc., a clock laneCL for transmitting clock signals, and a camera control interface CCIfor transmitting control signals, for example, between the videotransmitting apparatus 100 and the video receiving apparatus 200. ThoughFIG. 1 illustrates an example in which one data lane DL is provided, aplurality of data lanes DL may be provided. The camera control interfaceCCI includes a bidirectional control interface compatible with the I²C(Inter-Integrated Circuit) standards.

The video transmitting apparatus 100 includes an apparatus for sendingout signals according to the MIPI CSI-2 standards, the MIPI CSI-3standards, or the MIPI DSI standards. The video transmitting apparatus100 has a CSI transmitter 100A and a CCI slave 100B. The video receivingapparatus 200 has a CSI receiver 200A and a CCI master 200B. In theclock lane CL, the CSI transmitter 100A and the CSI receiver 200A areconnected to each other by a clock signal line. In the data lane DL, theCSI transmitter 100A and the CSI receiver 200A are connected to eachother by a clock signal line. In the camera control interface CCI, theCCI slave 100B and the CCI master 200B are connected to each other by acontrol signal line.

The CSI transmitter 100A includes a differential signal transmittingcircuit for generating a differential clock signal as a clock signal andoutputting the generated differential clock signal to the clock signalline. The CSI transmitter 100A may not necessarily transmit adifferential signal, but may transmit a single-ended or three-phasesignal. The CSI transmitter 100A also includes a differential signaltransmitting circuit for generating a differential data signal as a datasignal and outputting the generated differential data signal to the datasignal line, for example. The CSI receiver 200A includes a differentialsignal receiving circuit for receiving a differential clock signal as aclock signal and performing a predetermined processing process on thereceived differential clock signal. The CSI receiver 200A also includesa differential signal receiving circuit for receiving a differentialdata signal as a data signal and performing a predetermined processingprocess on the received differential data signal.

(Video Transmitting Apparatus 100)

FIG. 2 illustrates a configurational example of the video transmittingapparatus 100. The video transmitting apparatus 100 corresponds to aspecific example of the CSI transmitter 100A. The video transmittingapparatus 100 includes an image capturing section 110, image processingsections 120 and 130, and a transmitting section 140, for example. Thevideo transmitting apparatus 100 transmits transmission data 147Agenerated by performing a predetermined processing process on a capturedimage 111 obtained by the image capturing section 110 through the dataline DL to the video receiving apparatus 200. FIG. 3 illustrates anexample of a procedure for generating the transmission data 147A.

The image capturing section 110 converts an optical image obtainedthrough an optical lens into image data, for example. The imagecapturing section 110 includes a CCD (Charge Coupled Device) imagesensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.The image capturing section 110 has an analog-to-digital convertingcircuit that converts analog image data into digital image data. Theconverted image data may be of a YCbCr data format that represents thecolors of pixels with a luminance component Y and color differencecomponents Cb and Cr, or may be of a RGB data format. The imagecapturing section 110 outputs the captured image 111 (digital imagedata) obtained by image capturing to the image processing section 120.

The image processing section 120 includes a circuit for performing apredetermined processing process on the captured image 111 input fromthe image capturing section 110. According to the presupposed technology1, the image processing section 120 performs a predetermined processingprocess on the captured image 111 input from the image capturing section110 in a case where a control signal instructing the image processingsection 120 to segment ROIs is input from the video receiving apparatus200 through the camera control interface CCI. However, the presupposedtechnology 1 is also applicable where the video transmitting apparatus100, i.e., the transmission side, gives an instruction as to coordinatesfor segmenting ROIs. In this case, the transmission side receivesinformation representing “persons” or “objects” to be acquired by ROIssent out from the reception side, and makes a decision and gives aninstruction as to segmenting coordinates, for example. The videoreceiving apparatus 200 thus generates various kinds of data (120A, 120Band 120C) and outputs them to the transmitting section 140. The imageprocessing section 130 includes a circuit for performing a predeterminedprocessing process on the captured image 111 input from the imagecapturing section 110. The image processing section 130 performs apredetermined processing process on the captured image 111 input fromthe image capturing section 110 in a case where a control signalinstructing the image processing section 130 to output normal images isinput from the video receiving apparatus 200 through the camera controlinterface CCI. The image processing section 130 thus generates imagedata 130A and outputs them to the transmitting section 140.

The image processing section 130 has an encoding section 131, forexample. The encoding section 131 encodes the captured image 111 togenerate compressed image data 130A. The image processing section 130compresses the captured image 111 in a compression format that conformsto the JPEG (Joint Photographic Experts Group) standards, for example,as the format of the compressed image data 130A.

The image processing section 120 has a ROI segmenting section 121, a ROIanalyzing section 122, an overlap detecting section 123, a prioritysetting section 124, an encoding section 125, and an image processingcontrolling section 126, for example.

The ROI segmenting section 121 specifies an image or a plurality ofimages as an imaging target or targets included in the captured image111 input from the image capturing section 110, and establishes a regionof interest ROI per specified object. A region of interest ROI refers toa square-shaped region including a specified object, for example. TheROI segmenting section 121 specifies an image of each region of interestROI (for example, a ROI image 112 in FIG. 3) from the captured image111. The ROI segmenting section 121 further assigns a region number asan identifier to each established region of interest ROI. For example,in a case where the ROI segmenting section 121 has established tworegions of interest ROI in the captured image 111, the ROI segmentingsection 121 assigns a region number 1 to one of the regions of interestROI (for example, a region of interest ROI1 in FIG. 3) and assigns aregion number 2 to the other region of interest ROI (for example, aregion of interest ROI2 in FIG. 3). The ROI segmenting section 121stores the assigned identifiers (region numbers) in a storage section,for example. For example, the ROI segmenting section 121 stores each ROIimage 112 segmented from the captured image 111 in the storage section.Furthermore, for example, the ROI segmenting section 121 stores theidentifier (region number) assigned to each region of interest ROI, inthe storage section in association with the ROI image 112.

The ROI analyzing section 122 derives positional information 113 of eachregion of interest ROI in the captured image 111. The positionalinformation 113 includes, for example, the left upper end coordinates(Xa, Ya) of the region of interest ROI, the length in an X-axisdirection of the region of interest ROI, and the length in a Y-axisdirection of the region of interest ROI. The length in the X-axisdirection of the region of interest ROI refers, for example, to thephysical region length XLa in the X-axis direction of the region ofinterest ROI. The length in the Y-axis direction of the region ofinterest ROI refers, for example, to the physical region length YLa inthe Y-axis direction of the region of interest ROI. The physical regionlength represents the physical length, i.e., data length, of the regionof interest ROI. The positional information 113 may include thecoordinates of a position different from the left upper end of theregion of interest ROI. The ROI analyzing section 122 stores the derivedpositional information in the storage section, for example. The ROIanalyzing section 122 stores the derived positional information in thestorage section in association with the identifier, i.e., region number,assigned to the region of interest ROI.

The ROI analyzing section 122 may further derive, as the positionalinformation 113 per region of interest ROI, the output region length XLcin the X-axis direction of the region of interest ROI and the outputregion length YLc in the Y-axis direction of the region of interest ROI,for example. The output region length represents the physical length,i.e., data length, of the region of interest ROI after the resolution ofthe region of interest ROI has been changed by a decimating process oran addition of pixels, for example. The ROI analyzing section 122 mayderive, for example, as the positional information 113 per region ofinterest ROI, sensing information, exposure information, gaininformation, AD (Analog-Digital) word length, image format, etc., forexample, and store them in the storage section.

The sensing information refers to the contents of calculations aboutobjects included in the region of interest ROI and additionalinformation of a subsequent signal processing process on the ROI image112. The exposure information refers to an exposure time of the regionof interest ROI. The gain information refers to gain information of theregion of interest ROI. The AD word length refers to the word length ofdata per pixel AD-converted in the region of interest ROI. The imageformat refers to the format of the image of the region of interest ROI.The ROI analyzing section 122 may, for example, derive the number ofregions of interest ROI (the number of ROIs) included in the capturedimage 111 and store the number of ROIs in the storage section.

When a plurality of objects is specified as imaging targets in thecaptured image 111, the overlap detecting section 123 detects a regionof overlap (ROO (Region of Overlap)) where two or more regions ofinterest ROI overlap each other on the basis of the positionalinformation 113 of a plurality of regions of interest ROI in thecaptured image 111. Specifically, the overlap detecting section 123derives positional information 114 of each region of overlap ROO in thecaptured image 111. The overlap detecting section 123 stores the derivedpositional information 114 in the storage section, for example. Forexample, the overlap detecting section 123 stores the derived positionalinformation 114 in the storage section in corresponding relation to theregion of overlap ROO. The region of overlap ROO refers to asquare-shaped region identical or smaller in size to the smallest regionof interest ROI among two or more regions of interest ROI that overlapeach other. The positional information 114 includes, for example, theleft upper end coordinates (Xb, Yb) of the region of overlap ROO, thelength in the X-axis direction of the region of overlap ROO, and thelength in the Y-axis direction of the region of overlap ROO. The lengthin the X-axis direction of the region of overlap ROO refers, forexample, to the physical region length XLb. The length in the Y-axisdirection of the region of overlap ROO refers, for example, to thephysical region length YLb. The positional information 114 may includethe coordinates of a position different from the left upper end of theregion of interest ROI.

The priority setting section 124 assigns a priority 115 to each regionof interest ROI in the captured image 111. The priority setting section124 stores the assigned priority 115 in the storage section, forexample. For example, the priority setting section 124 stores theassigned priority 115 in the storage section in corresponding relationto the region of interest ROI. The priority setting section 124 mayassign a priority 115 to each region of interest ROI separately from theregion number assigned to each region of interest ROI, or may use theregion number assigned to each region of interest ROI instead of apriority 115. The priority setting section 124 may, for example, storethe priority 115 in the storage section in association with the regionof interest ROI or may store the region number assigned to each regionof interest ROI in the storage section in association with the region ofinterest ROI.

The priority 115 refers to an identifier of each region of interest ROI,and represents discriminating information for discriminating which oneof a plurality of regions of interest ROI in the captured image 111 aregion of overlap ROO has been eliminated from. For example, thepriority setting section 124 assigns “1” as a priority 115 to one of tworegions of interest ROI each including a region of overlap ROO andassigns “2” as a priority 115 to the other region of interest ROI. Inthis case, a region of overlap ROO is eliminated with respect to aregion of interest ROI where the numerical value of the priority 115 islarger in generating a transmission image 116 to be described later.Incidentally, the priority setting section 124 may assign the samenumber as the region number assigned to each region of interest ROI as apriority 115 to the region of interest ROI. For example, the prioritysetting section 124 stores the priority 115 assigned to each region ofinterest ROI in the storage section in association with the ROI image112.

The encoding section 125 encodes each transmission image 116 to generatecompressed image data 120A. The encoding section 125 compresses eachtransmission image 116 in a compression format that conforms to the JPEGstandards, for example, as the format of the compressed image data 120A.Before performing the above compression process, the encoding section125 generates each transmission image 116. In order that an image 118 ofa region of overlap ROO will not overlappingly be included in aplurality of ROI images 112 obtained from the captured image 111, theencoding section 125 generates a plurality of transmission images 116where the image 118 has been eliminated from the plurality of ROI images112 obtained from the captured image 111.

The encoding section 125 determines which one of a plurality of ROIimages 112 the image 118 is to be eliminated from, on the basis of thepriority 115 assigned to each region of interest ROI, for example. Theencoding section 125 may determine, for example, which one of aplurality of ROI images 112 the image 118 is to be eliminated from, byusing the region number assigned to each region of interest ROI as apriority 115. The encoding section 125 uses the ROI image 112 asspecified above from which the image 118 has been eliminated as atransmission image 116 (for example, a transmission image 116 a 2 inFIG. 3). The encoding section 125 uses the ROI image 112 that does notinclude a region of overlap ROO or the ROI image 112 which the image 118has not been eliminated from as determined above, as a transmissionimage 116 (for example, a transmission image 116 a 1 in FIG. 3).

The image processing controlling section 126 generates ROI information120B and frame information 120C and transmits them to the transmittingsection 140. The ROI information 120B includes each positionalinformation 113, for example. Furthermore, the ROI information 120Bincludes at least one of the data type of each region of interest ROI,the number of regions of interest ROI included in the captured image111, the region number (or the priority 115) of each region of interestROI, the data length of each region of interest ROI, and the imageformat of each region of interest ROI. The frame information 120Cincludes the number of a virtual channel assigned to each frame, thedata type of each region of interest ROI, the payload length per line,etc., for example. The data type includes YUV data, RGB data, or RAWdata, for example. Furthermore, the data type includes data of the ROIformat, data of the normal format, etc., for example. The payload lengthrepresents the number of pixels included in the payload of a longpacket, e.g., the number of pixels per region of interest ROI. Thepayload refers to major data (application data) transmitted between thevideo transmitting apparatus 100 and the video receiving apparatus 200.The long packet refers to a packet disposed between a packet header PHand a packet footer PF.

The transmitting section 140 includes a circuit for generating andsending out transmission data 147A on the basis of various kinds of data(data 120A, 120B, 120C and 130A) input from the image processingsections 120 and 130. The transmitting section 140 sends out the ROIinformation 120B regarding each region of interest ROI in the capturedimage 111 as embedded data. Furthermore, in a case where a controlsignal indicating the segmentation of ROIs is input from the videoreceiving apparatus 200 via the camera control interface CCI, thetransmitting section 140 sends out the image data (compressed image data120A) of each region of interest ROI as the payload data of a longpacket. At this time, the transmitting section 140 sends out the imagedata (compressed image data 120A) of each region of interest ROI in acommon virtual channel. Furthermore, the transmitting section 140 sendsout the image data (compressed image data 120A) of each region ofinterest ROI as an image data frame, and sends out the ROI information120B regarding each region of interest ROI as the header of an imagedata frame. Furthermore, in a case where a control signal indicating theoutputting of a normal image is input from the video receiving apparatus200 via the camera control interface CCI, the transmitting section 140sends out normal image data (compressed image data 130A) as the payloaddata of a long packet.

The transmitting section 140 has a LINK controlling section 141, an ECCgenerating section 142, a PH generating section 143, an EBD buffer 144,a ROI data buffer 145, a normal image data buffer 146, and a combiningsection 147. In a case where a control signal indicating thesegmentation of ROIs is input from the video receiving apparatus 200 viathe camera control interface CCI, the LINK controlling section 141, theECC generating section 142, the PH generating section 143, the EBDbuffer 144, and the ROI data buffer 145 output data to the combiningsection 147. In a control signal indicating the outputting of a normalimage is input from the video receiving apparatus 200 via the cameracontrol interface CCI, the normal image data buffer 146 outputs data tothe combining section 147.

It is noted that the ROI data buffer 145 may doubles as the normal imagedata buffer 146. In this case, the transmitting section 140 may have aselector for selecting the output from either one of the ROI data buffer145 and the ROI data buffer 145, between the output terminals of the ROIdata buffer 145 and the ROI data buffer 145 and an input terminal of thecombining section 147.

The LINK controlling section 141 outputs the frame information 120C perline to the LINK controlling section 141 and the ECC generating section142, for example. The ECC generating section 142 generates an errorcorrecting code for a line in the frame information 120C, for example,on the basis of the data of the line, e.g., the number of the virtualchannel, the data type of each region of interest ROI, the payloadlength per line, etc. The ECC generating section 142 outputs thegenerated error correcting code to the PH generating section 143, forexample. The PH generating section 143 generates a packet header PH perline using the frame information 120C and the error correcting codegenerated by the ECC generating section 142, for example. At this time,as illustrated in FIG. 4, for example, the packet header PH includes apacket header of the payload data of a long packet. The packet header PHincludes DI, WC, and ECC, for example. WC represents an area forindicating the end of a packet with the number of words to the videoreceiving apparatus 200. WC includes a payload length, for example, andincludes the number of pixels per region of interest ROI, for example.ECC represents an area for storing a value for correcting a bit error.ECC includes an error correcting code. DI represents an area for storinga data identifier. DI includes the number of a VC (virtual channel) andDataType (data type of each region of interest ROI). VC (virtualchannel) refers to a concept introduced for flow control of packets andrepresents a mechanism for supporting a plurality of independent datastreams that share one link. The PH generating section 143 outputs thegenerated packet header PH to the combining section 147.

The EBD buffer 144 primarily stores ROI information 120B and outputs theROI information 120B as embedded data to the combining section 147. Theembedded data refer to additional information that can be embedded inthe header or footer of an image data frame (see FIG. 5 to be describedlater). The embedded data include ROI information 120B, for example.

The ROI data buffer 145 primarily stores compressed image data 120A andoutputs the compressed image data 120A at predetermined timing as thepayload data of a long packet to the combining section 147. In a casewhere a control signal indicating the segmentation of ROIs is input fromthe video receiving apparatus 200 via the camera control interface CCI,the ROI data buffer 145 outputs the compressed image data 120A as thepayload data of a long packet to the combining section 147. The normalimage data buffer 146 primarily stores compressed image data 130A andoutputs the compressed image data 130A at predetermined timing as thepayload data of a long packet to the combining section 147. In a casewhere a control signal indicating the outputting of a normal image isinput from the video receiving apparatus 200 via the camera controlinterface CCI, the normal image data buffer 146 outputs the compressedimage data 130A as the payload data of a long packet to the combiningsection 147.

In a case where a control signal indicating the outputting of a normalimage is input from the video receiving apparatus 200 via the cameracontrol interface CCI, the combining section 147 generates transmissiondata 147A on the basis of input data (compressed image data 130A). Thecombining section 147 outputs the generated transmission data 147A tothe video receiving apparatus 200 via the data lane DL. On the otherhand, in a case where a control signal indicating the segmentation ofROIs is input from the video receiving apparatus 200 via the cameracontrol interface CCI, the combining section 147 generates transmissiondata 147A on the basis of various input data (a packet header PH, ROIinformation 120B, and compressed image data 120A). The combining section147 outputs the generated transmission data 147A to the video receivingapparatus 200 via the data lane DL. Specifically, the combining section147 includes DataType (data type of each region of interest ROI) in thepacket header PH of the payload data of a long packet and sends out thedata. Furthermore, the combining section 147 sends out image data(compressed image data 120A) of each region of interest ROI in a commonvirtual channel.

The transmission data 147A include an image data frame as illustrated inFIG. 5, for example. The image data frame normally has a header area, apacket area, and a footer area. In FIG. 5, the footer area is omittedfrom illustration for the sake of convenience. The frame header area R1of the transmission data 147A includes embedded data. At this time, theembedded data include ROI information 120B. In FIG. 5, the packet areaR2 of the transmission data 147A includes the payload data of a longpacket per line, and also include a packet header PH and a packet footerPF at positions sandwiching the payload data of a long packet.Furthermore, the packet area R2 includes low power modes LP at positionssandwiching the packet header PH and the packet footer PF.

At this time, the packet header PH includes DI, WC, and ECC, forexample. WC includes a payload length, for example, and includes thenumber of pixels per region of interest ROI, for example. ECC includesan error correcting code. DI includes the number of a VC (virtualchannel) and DataType (data type of each region of interest ROI).According to the present embodiment, the number of a common virtualchannel is assigned to a VC of each line. In FIG. 5, the packet area R2of the transmission data 147A includes compressed image data 147B. Thecompressed image data 147B includes one compressed image data 120A or aplurality of compressed image data 120A. Here in FIG. 5, a packet groupcloser to the packet header PH includes compressed image data 120A(120A1) of the transmission image 116 a 1 in FIG. 3, and a packet groupremoter from the packet header PH includes compressed image data 120A(120A2) of the transmission image 116 a 2 in FIG. 3. These twocompressed image data 120A1 and 120A2 make up the compressed image data147B. The payload data of a long packet of each line include one line ofpixel data in the compressed image data 147B.

FIG. 6 illustrates a configurational example of the transmission data147A. The transmission data 147A include a frame header area R1 and apacket area R2, for example. Incidentally, FIG. 6 illustrates details ofthe contents of the frame header area R1. Furthermore, low power modesLP are omitted from illustration in FIG. 6.

The frame header area R1 includes a frame number F1 as an identifier ofthe transmission data 147A, for example. The frame header area R1includes information regarding compressed image data 147B included inthe packet area R2. The frame header area R1 includes, for example, thenumber of compressed image data 120A (the number of ROIs) included inthe compressed image data 147B and information regarding the ROI image112 (ROI information 120B) corresponding to each compressed image data120A included in the compressed image data 147B.

The combining section 147 divides and places compressed image data 147Bper pixel row of compressed image data 120A in the packet area R2 of thetransmission data 147A, for example. Therefore, the packet area R2 ofthe transmission data 147A does not include overlapping compressed imagedata corresponding to an image 118 of a region of overlap ROO.Furthermore, the combining section 147 has eliminated therefrom a pixelrow not corresponding to each transmission image 116 of the capturedimage 111 in the packet area R2 of the transmission data 147A, forexample. Consequently, the packet area R2 of the transmission data 147Adoes not include a pixel row not corresponding to each transmissionimage 116 of the captured image 111. Incidentally, in the packet area R2in FIG. 6, a zone surrounded by the broken line corresponds tocompressed image data of an image 118 of a region of overlap ROO.

The boundary between a packet group closer to the packet header PH (forexample, 1(n) in FIG. 6) and a packet group remoter from the packetheader PH (for example, 2(1) in FIG. 6) is specified by the physicalregion length XLa1 of the ROI image 112 corresponding to the compressedimage data of the packet group closer to the packet header PH (forexample, 1(n) in FIG. 6). A packet starting position in the compressedimage data corresponding to an image 118 of a region of overlap ROOincluded in a packet group closer to the packet header PH (for example,1(n) in FIG. 6) is specified by the physical region length XLa2 of theROI image 112 corresponding to a packet group remoter from the packetheader PH (for example, 2(1) in FIG. 6).

When the payload data of a long packet is to be generated per line inthe packet area R2 of the transmission data 147A, for example, thecombining section 147 may include ROI information 120B, as illustratedin FIG. 7, for example, other than pixel data of one line in thecompressed image data 147B, in the payload data of the long packet. Inother words, the combining section 147 may include ROI information 120Bin the payload data of a long packet and output the data. At this time,as illustrated in FIG. 7(A) to FIG. 7(K), the ROI information 120Bincludes at least one of the number of regions of interest ROI (thenumber of ROIs) included in the captured image 111, the region number(or the priority 115) of each region of interest ROI, the data length ofeach region of interest ROI, and the image format of each region ofinterest ROI. The ROI information 120B should preferably be placed inthe payload data of a long packet at the end on the packet header PHside (i.e., the leading end of the payload data of the long packet).

(Video Receiving Apparatus 200)

Next, the video receiving apparatus 200 will be described below. FIG. 8illustrates a configurational example of the video receiving apparatus200. FIG. 9 illustrates an example of a procedure for generating a ROIimage 223A in the video receiving apparatus 200. The video receivingapparatus 200 includes an apparatus for receiving signals according tostandards common to the video transmitting apparatus 100 (for example,the MIPI CSI-2 standards, the MIPI CSI-3 standards, or the MIPI DSIstandards). The video receiving apparatus 200 has a receiving section210 and an information processing section 220. The receiving section 210includes a circuit for receiving transmission data 147A output from thevideo transmitting apparatus 100 via the data lane DL, performing apredetermined process on the received transmission data 147A to generatevarious kinds of data (214A, 215A and 215B), and outputting thegenerated data to the information processing section 220. Theinformation processing section 220 includes a circuit for generating aROI image 223A based on various kinds of data (214A and 215A) receivedfrom the receiving section 210 and generating a normal image 224A basedon data (215B) received from the receiving section 210.

The receiving section 210 has, for example, a header separating section211, a header interpreting section 212, a payload separating section213, an EBD interpreting section 214, and a ROI data separating section215.

The header separating section 211 receives transmission data 147A fromthe video transmitting apparatus 100 via the data lane DL. Specifically,the header separating section 211 receives transmission data 147Aincluding ROI information 120B regarding each region of interest ROI inthe captured image 111 in embedded data and also including image data(compressed image data 120A) of each region of interest ROI in thepayload data of a long packet. The header separating section 211separates the received transmission data 147A into a frame header areaR1 and a packet area R2. The header interpreting section 212 specifiesthe positions of the payload data of long packets included in the packetarea R2 on the basis of data (specifically, embedded data) included inthe frame header area R1. The payload separating section 213 separatesthe payload data of the long packets included in the packet area R2 fromthe packet area R2 on the basis of the positions of the payload data ofthe long packets that have been specified by the header interpretingsection 212.

The EBD interpreting section 214 outputs the embedded data as EBD data214A to the information processing section 220. Furthermore, the EBDinterpreting section 214 discriminates whether the image data includedin the payload data of the long packets are the compressed image data120A of the image data 116 of a ROI or the compressed image data 130A ofnormal image data, from the data type included in the embedded data. TheEBD interpreting section 214 outputs the discriminated result to the ROIdata separating section 215.

If the image data included in the payload data of the long packets arethe compressed image data 120A of the image data 116 of a ROI, then theROI data separating section 215 outputs the payload data of the longpacket as payload data 215A to the information processing section 220(specifically, a ROI decoding section 222). If the image data includedin the payload data are the compressed image data 130A, then the ROIdata separating section 215 outputs the payload data of the long packetas payload data 215A to the information processing section 220(specifically, a normal image decoding section 224). In a case where thepayload data of the long packet include the ROI information 120B, thepayload data 215A include the ROI information 120B and one line of pixeldata of the compressed image data 147B.

The information processing section 220 extracts the ROI information 120Bfrom the embedded data included in the EBD data 214A. The informationprocessing section 220 extracts an image of each region of interest ROI(ROI image 112) in the captured image 111 from the payload data of thelong packet included in the transmission data 147A received by thereceiving section 210 on the basis of the ROI information 120B extractedby an information extracting section 221. The information processingsection 220 has, for example, the information extracting section 221,the ROI decoding section 222, a ROI image generating section 223, andthe normal image decoding section 224.

The normal image decoding section 224 decodes the payload data 215B togenerate a normal image 224A. The ROI decoding section 222 decodes thecompressed image data 147B included in the payload data 215A to generateimage data 222A. The image data 222A represent one transmission image116 or a plurality of transmission images 116.

The information extracting section 221 extracts the ROI information 120Bfrom the embedded data included in the EBD data 214A. For example, theinformation extracting section 221 extracts the number of regions ofinterest ROI included in the captured image 111, the region number (orthe priority 115) of each region of interest ROI, the data length ofeach region of interest ROI, and the image format of each region ofinterest ROI, for example, from the embedded data included in the EBDdata 214A. In other words, the transmission data 147A include the regionnumber (or the priority 115) of a region of interest ROI correspondingto each transmission image 116 as discriminating information fordiscriminating which one of a plurality of transmission images 116obtained from the transmission data 147A an image 118 of a region ofoverlap ROO has been eliminated from.

The ROI image generating section 223 detects a region of overlap ROOwhere two or more regions of interest ROI overlap each other on thebasis of the ROI information 120B obtained by the information extractingsection 221.

The information extracting section 221 extracts, for example,coordinates (for example, left upper end coordinates (Xa1, Ya1)),lengths (for example, physical region lengths XLa1 and YLa1), and aregion number 1 (or a priority 115 (=1)) of a region of interest ROIcorresponding to a ROI image 112 a 1 from the embedded data included inthe EBD data 214A. Furthermore, the information extracting section 221extracts, for example, coordinates (for example, left upper endcoordinates (Xa2, Ya2)), lengths (for example, physical region lengthsXLa2, YLa2), and a region number 2 (or a priority 115 (=2)) of a regionof interest ROI corresponding to a ROI image 112 a 2 from the embeddeddata included in the EBD data 214A.

At this time, the ROI image generating section 223 derives positionalinformation 114 of the region of overlap ROO based on these extractedpieces of information (hereinafter referred to as “extracted information221A”). The ROI image generating section 223 derives, for example,coordinates (for example, left upper end coordinates Xb1, Yb1) andlengths (for example, physical region lengths XLb1 and YLb1) of theregion of overlap ROO as the positional information 114 of the region ofoverlap ROO.

Incidentally, the ROI image generating section 223 may acquire the ROIinformation 120B from the payload data 215A instead of acquiring the ROIinformation 120B from the embedded data included in the EBD data 214A.In this case, the ROI image generating section 223 may detect a regionof overlap ROO where two or more regions of interest ROI overlap eachother on the basis of the ROI information 120B included in the payloaddata 215A. Furthermore, the ROI image generating section 223 may extractthe extracted information 221A from the ROI information 120B included inthe payload data 215A, and may derive the positional information 114 ofa region of overlap ROO based on the extracted information 221A thusextracted.

Moreover, the ROI image generating section 223 generates an image (ROIimages 112 a 1 and 112 a 2) of each region of interest ROI in thecaptured image 111 on the basis of the image data 222A, the extractedinformation 221A, and the positional information 114 of the region ofoverlap ROO. The ROI image generating section 223 outputs the generatedimages as a ROI image 223A.

[Procedure]

Next, an example of a procedure for transmitting data in the videotransmission system 1 will be described below with reference to FIGS. 3and 9.

First, the image capturing section 110 outputs a captured image 111(digital image data) obtained by image capturing to the image processingsection 120. The ROI segmenting section 121 specifies two regions ofinterest ROI1 and ROI2 included in the captured image 111 input from theimage capturing section 110. The ROI segmenting section 121 segmentsimages of the respective regions of interest ROI1 and ROI2 (ROI images112 a 1 and 112 a 2) from the captured image 111. The ROI segmentingsection 121 assigns a region number 1 as an identifier to the region ofinterest ROI1 and assigns a region number 2 as an identifier to theregion of interest ROI2.

The ROI analyzing section 122 derives positional information 113 of eachregion of interest ROI in the captured image 111. The ROI analyzingsection 122 derives left upper coordinates (Xa1, Ya1) of the region ofinterest ROI1, a length (XLa1) in the X-axis direction of the region ofinterest ROI1, and a length (YLa1) in the Y-axis direction of the regionof interest ROI1 on the basis of the region of interest ROI1. The ROIanalyzing section 122 derives left upper coordinates (Xa2, Ya2) of theregion of interest ROI2, a length (XLa2) in the X-axis direction of theregion of interest ROI2, and a length (YLa2) in the Y-axis direction ofthe region of interest ROI2 on the basis of the region of interest ROI2.

The overlap detecting section 123 detects a region of overlap ROO wherethe two regions of interest ROI1 and ROI2 overlap each other on thebasis of the positional information 113 of the two regions of interestROI1 and ROI2 in the captured image 111. Specifically, the overlapdetecting section 123 derives positional information 114 of the regionof overlap ROO in the captured image 111. The overlap detecting section123 derives left upper coordinates (Xb1, Yb1) of the region of overlapROO, a length (XLb1) in the X-axis direction of the region of overlapROO, and a length (YLb1) in the Y-axis direction of the region ofoverlap ROO as the positional information 114 of the region of overlapROO in the captured image 111.

The priority setting section 124 assigns “1” as a priority 115 to theregion of interest ROI1 that is one of the two regions of interest ROI1and ROI2, and assigns “2” as a priority 115 to the other region ofinterest ROI2.

The encoding section 125 generates two transmission images 116 a 1 and116 a 2 where an image 118 of the region of overlap ROO has beeneliminated from the two ROI images 112 a 1 and 112 a 2 obtained from thecaptured image 111, in order that the image 118 will not overlappinglybe included in the two regions of interest ROI1 and ROI2.

The encoding section 125 determines which one of the two ROI images 112a 1 and 112 a 2 the image 118 is to be eliminated from on the basis ofregion numbers (or the priority 115) of the two regions of interest ROI1and ROI2. The encoding section 125 eliminates the image 118 from the ROIimage 112 a 2 corresponding to the region of interest ROI2 whose regionnumber (or the priority 115) is larger among the two regions of interestROI1 and ROI2, thereby generating a transmission image 116 a 2. Theencoding section 125 uses the ROI image 112 a 1 itself corresponding tothe region of interest ROI1 whose region number (or the priority 115) issmaller among the two regions of interest ROI1 and ROI2, as atransmission image 116 a 1.

The image processing controlling section 126 generates ROI information120B and frame information 120C and transmits them to the transmittingsection 140. The transmitting section 140 generates transmission data147A based on various kinds of data (120A, 120B, 120C and 130A) inputfrom the image processing sections 120 and 130. The transmitting section140 sends out the generated transmission data 147A to the videoreceiving apparatus 200 via the data lane DL.

The receiving section 210 receives the transmission data 147A outputfrom the video transmitting apparatus 100 via the data lane DL. Thereceiving section 210 performs a predetermined process on the receivedtransmission data 147A to generate EBD data 214A and payload data 215Aand outputs them to the information processing section 220.

The information extracting section 221 extracts ROI information 120Bfrom the embedded data included in the EBD data 214A. The informationextracting section 221 extracts coordinates (for example, left upper endcoordinates (Xa1, Ya1)), lengths (for example, physical region lengthsXLa1 and YLa1), and a region number 1 (or a priority 115 (=1)) of theregion of interest ROI corresponding to the ROI image 112 a 1 from theembedded data included in the EBD data 214A. Furthermore, theinformation extracting section 221 extracts coordinates (for example,left upper end coordinates (Xa2, Ya2)), lengths (for example, physicalregion lengths XLa2, YLa2), and a region number 2 (or a priority 115(=2)) of the region of interest ROI corresponding to the ROI image 112 a2 from the embedded data included in the EBD data 214A. The ROI decodingsection 222 decodes the compressed image data 147B included in thepayload data 215A to generate image data 222A.

The ROI image generating section 223 derives the positional information114 of the region of overlap ROO based on the extracted pieces ofinformation (extracted information 221A). The ROI image generatingsection 223 extracts, for example, coordinates (for example, left upperend coordinates Xb1, Yb1) and lengths (for example, physical regionlengths XLb1 and YLb1) of the region of overlap ROO as the positionalinformation 114 of the region of overlap ROO. Furthermore, the ROI imagegenerating section 223 generates an image (ROI images 112 al and 112 a2) of each region of interest ROI in the captured image 111 on the basisof the image data 222A, the extracted information 221A, and thepositional information 114 of the region of overlap ROO.

[Advantages]

Next, advantages of the video transmission system 1 according to thepresent embodiment will be described below.

In recent years, there have been growing applications in which largeamounts of data are transmitted in bulk. Such applications tend to poselarge loads on the transmission system, possibly causing thetransmission system to go down in worst-case scenarios and fail toperform data transmission.

To avoid transmission system shutdowns, it has customary in the art tospecify an object as an imaging target and transmit only a partial imageof the specified object that has been segmented, rather thantransmitting an entire captured image.

Incidentally, MIPI CS1-2 may be used as a process of transmitting datafrom an image sensor to an application sensor. It may not be easy totransmit ROIs according to this process due to various limitations.

On the other hand, according to the present embodiment, ROI information120B regarding each region of interest ROI in the captured image 111 issent out as embedded data, and image data of each region of interest ROIare sent out as the payload data of a long packet. Therefore, anapparatus (video receiving apparatus 200) that has received transmissiondata 147A sent out from the video transmitting apparatus 100 can easilyextract the image data (ROI image 112) of each region of interest ROIfrom the transmission data 147A. As a result, it is possible to transmitregions of interest ROIs regardless of various limitations.

According to the present embodiment, furthermore, the image data(compressed image data 120A) of each region of interest ROI are sent outin a common virtual channel. Since a plurality of ROI images 112 canthus be sent in one packet, it is not necessary to enter an LP modewhile the plurality of ROI images 112 is being sent, resulting in a hightransmission efficiency.

According to the present embodiment, moreover, a data type of eachregion of interest ROI is included in the packet header PH of thepayload data of the long packet and sent. Therefore, the data type ofeach region of interest ROI can be obtained simply by accessing thepacket header PH of the payload data of the long packet, rather thanaccessing the embedded data. Inasmuch as this increases the processingrate of the video receiving apparatus 200, a high transmissionefficiency can be achieved.

According to the present embodiment, furthermore, in a case where theROI information 120B is included in the payload data of a long packetand sent, the ROI information 120B can be obtained simply by accessingthe payload data of the long packet, rather than accessing the embeddeddata. Inasmuch as this increases the processing rate of the videoreceiving apparatus 200, a high transmission efficiency can be achieved.

According to the present embodiment, moreover, the ROI information 120Bregarding each region of interest ROI is extracted from the embeddeddata included in the transmission data 147A and an image of each regionof interest ROI (ROI image 112) is extracted from the payload data ofthe long packet include in the transmission data 147A on the basis ofthe extracted ROI information 120B. This allows the image of each regionof interest ROI (ROI image 112) to be easily extracted from thetransmission data 147A. As a result, it is possible to transmit regionsof interest ROIs regardless of various limitations.

2. Technology 2 that is Presupposed for the Present Disclosure

A technology for transmitting a region of interest (ROI) as a partialregion (non-rectangular in shape) segmented from a captured image willbe described below using FIGS. 10 through 12 with reference to FIGS. 1through 9. Specifically, a technology for transmitting and receiving animage of an object as an imaging target that is of a shape other than asquare shape (rectangular shape) will be described below. FIG. 10 is adiagram schematically illustrating regions where objects specified in acaptured image 111 are placed. For an easier understanding, FIG. 10depicts the captured image 111 that is captured in an image capturingregion including image capturing elements arranged in 15 rows×23columns. FIG. 11 is a diagram illustrating an example of ROIsestablished with respect to the specified objects.

According to the presupposed technology 2, as with the presupposedtechnology 1, there will be described a situation where a predeterminedprocess is performed on the captured image 111 input from the imagecapturing section 110 in a case where a control signal indicating thesegmentation of ROIs is input from the video receiving apparatus 200 viathe camera control interface CCI to the video transmitting apparatus100. However, the presupposed technology 2 is also applicable to asituation where the video transmitting apparatus 100, i.e., thetransmission side, indicates coordinates for segmenting ROIs. In such acase, the transmission side is configured to receive informationrepresenting “persons” or “objects” to be acquired by ROIs sent out fromthe reception side, and to make a decision and give an instruction as tosegmenting coordinates, for example.

A control signal indicating the segmentation of ROIs is input from thevideo receiving apparatus 200 via the camera control interface CCI. Inresponse to the control signal, as illustrated in FIG. 10, the ROIsegmenting section 121 specifies four objects 1 through 4 included asimaging targets in the captured image 111. The object 1 has arectangular shape taking up a portion of a left upper region of thecaptured image 111, for example. The object 2 has a shape taking up apartial region on the right side of the object 1 in the captured image111 and devoid of both side corners of an upper side of a rectangularshape and a portion of a lower side thereof, for example. The object 3has a shape taking up a partial region below the object 2 in thecaptured image 111 and devoid of four corners of a rectangular shape,for example. The object 4 has a shape taking up a partial region belowthe object 3 in the captured image 111 and devoid of both side cornersof an upper side of a rectangular shape, for example. The object 3 andthe object 4 partly overlap each other.

As illustrated in FIG. 11, the ROI segmenting section 121 (see FIG. 2)establishes minimum rectangular shapes including the specified objectsas regions of interest ROI1 through ROI4, respectively. The ROIsegmenting section 121 establishes the region of interest ROI1 for theobject 1 and segments a ROI image 112 a 1. Furthermore, the ROIsegmenting section 121 establishes the region of interest ROI2 for theobject 2 and segments a ROI image 112 a 2. Furthermore, the ROIsegmenting section 121 establishes the region of interest ROI3 for theobject 3 and segments a ROI image 112 a 3. Furthermore, the ROIsegmenting section 121 establishes the region of interest ROI4 for theobject 4 and segments a ROI image 112 a 4.

The ROI segmenting section 121 stores the region of interest ROI1 and aregion number “1” assigned to the region of interest ROI1 in the storagesection in association with each other. The ROI segmenting section 121stores the region of interest ROI2 and a region number “2” assigned tothe region of interest ROI2 in the storage section in association witheach other. The ROI segmenting section 121 stores the region of interestROI3 and a region number “3” assigned to the region of interest ROI3 inthe storage section in association with each other. The ROI segmentingsection 121 stores the region of interest ROI4 and a region number “4”assigned to the region of interest ROI4 in the storage section inassociation with each other.

The ROI analyzing section 122 (see FIG. 2) derive positional informationof the respective regions of interest ROI1 through ROI4. The ROIanalyzing section 122 derives a physical region length XLa1 in theX-axis direction and a physical region length YLa1 in the Y-axisdirection, for example, as the positional information of the region ofinterest ROI1. The ROI analyzing section 122 derives a physical regionlength XLa2 in the X-axis direction and a physical region length YLa2 inthe Y-axis direction, for example, as the positional information of theregion of interest ROI2. The ROI analyzing section 122 derives aphysical region length XLa3 in the X-axis direction and a physicalregion length YLa3 in the Y-axis direction, for example, as thepositional information of the region of interest ROI3. The ROI analyzingsection 122 derives a physical region length XLa4 in the X-axisdirection and a physical region length YLa4 in the Y-axis direction, forexample, as the positional information of the region of interest ROI4.Furthermore, the ROI analyzing section 122 may derive, as positionalinformation 113 of each region of interest ROI, an output region lengthXLc in the X-axis direction of the region of interest ROI and an outputregion length YLc in the Y-axis direction of the region of interest ROI,for example.

The ROI analyzing section 122 derives sizes and total amounts of data ofthe respective regions of interest ROI1 through ROI4 as information fora subsequent stage by deriving the lengths in the X-axis direction andthe Y-axis directions of the respective regions of interest ROIs. Thevideo receiving apparatus 200 that represents the subsequent stage canthus secure a memory space.

The ROI analyzing section 122 is configured to derive positionalinformation of the ROI images 112 a 1 through 112 a 4, not thepositional information of the regions of interest ROI, in a case wherethe objects as imaging targets and the regions of interest do not agreewith each other in shape. The ROI analyzing section 122 derives left endcoordinates (xn, yn) and physical region lengths XLn in the X-axisdirection of the respective rows as the positional information of theROI images 112 a 1 through 112 a 4. Furthermore, in a case where a ROIimage is separated as in the second row of the ROI image 112 a 2, theROI analyzing section 122 derives respective positional information ofthe separated portions. The ROI analyzing section 122 stores the regionnumbers of the regions of interest ROI1 through ROI4 and the positionalinformation of the ROI images 112 a 1 through 112 a 4 in the storagesection in association with each other.

Moreover, the ROI analyzing section 122 may derive sensing information,exposure information, gain information, AD word length, image format,etc., for example, other than the positional information, of therespective regions of interest ROI1 through ROI4, and store them in thestorage section in association with the region numbers.

In a case where objects as imaging targets are of a rectangular shape,the overlap detecting section 123 (see FIG. 2) derives a region whereROI images overlap each other, not a region where regions of interestoverlap each other, as a region of overlap. As illustrated in FIG. 11,the overlap detecting section 123 derives a region of overlap ROO as aregion where the ROI image 112 a 3 and the ROI image 123 a 4 overlapeach other. The overlap detecting section 123 stores the derived regionof overlap ROO in the storage section in association with the respectivepositional information of the regions of interest ROI3 and ROI4.

The priority setting section 124 (see FIG. 2) assigns the priority “1”to the region of interest ROI1, and stores the priority “1” in thestorage section in association with the region of interest ROI1. Thepriority setting section 124 assigns the priority “2” that is lower thanthe priority “1” to the region of interest ROI2, and stores the priority“2” in the storage section in association with the region of interestROI2. The priority setting section 124 assigns the priority “3” that islower than the priority “2” to the region of interest ROI3, and storesthe priority “3” in the storage section in association with the regionof interest ROI3. The priority setting section 124 assigns the priority“4” that is lower than the priority “3” to the region of interest ROI4,and stores the priority “4” in the storage section in association withthe region of interest ROI4.

The encoding section 125 (see FIG. 2) generates respective transmissionimages of the ROI images 112 a 1 through 112 a 4. Since the priority ofthe region of interest ROI4 is lower than the priority of the region ofinterest ROI3, the encoding section 125 generates a transmission imageby eliminating the region of overlap ROO from the ROI image 112 a 4.

The image processing controlling section 126 (see FIG. 2) generates ROIinformation and frame information and transmits them to the transmittingsection 140 (see FIG. 2). The ROI information includes the respectivepositional information of the ROI images 112 a 1 through 112 a 4, forexample. The ROI information also includes, other than the positionalinformation, information (for example, the respective data types of theregions of interest ROI1 through ROI4, the number of the regions ofinterest ROI1 through ROI4 included in the captured image 111, theregion numbers and priority of the regions of interest ROI1 throughROI4, etc.) similar to those in a case where objects as imaging targetsare of a rectangular shape. The frame information includes, for example,information similar to those in a case where objects as imaging targetsare of a rectangular shape, such as data types of the regions ofinterest ROI1 through ROI4.

The LINK controlling section 141 provided in the transmitting section140 (see FIG. 2) outputs the frame information and the ROI informationinput from the image processing controlling section 126 per line to theECC generating section 142 and the PH generating section 143 (see FIG. 2for both). The ECC generating section 142 generates an error correctingcode for a line in the frame information on the basis of data of theline (for example, the number of the virtual channel, the respectivedata types of the regions of interest ROI1 through ROI4, the payloadlength per line, etc.), for example. The ECC generating section 142outputs the generated error correcting code to the PH generating section143, for example. The PH generating section 143 generates a packetheader PH (see FIG. 4) per line, using the frame information and theerror correcting code generated by the ECC generating section 142.

The EBD buffer 144 (see FIG. 2) primarily stores the ROI information andoutputs the ROI information at predetermined timing as embedded data tothe combining section 147 (see FIG. 2).

The ROI data buffer 145 (see FIG. 2) primarily stores the compressedimage data input from the encoding section 125 and outputs thecompressed image data 120A as the payload data of a long packet to thecombining section 147 in a case where a control signal indicating thesegmentation of ROIs is input from the video receiving apparatus 200 viathe camera control interface CCI.

In a case where a control signal indicating the segmentation of ROIs isinput from the video receiving apparatus 200 via the camera controlinterface CCI, the combining section 147 generates transmission data147A based on various input data (the packet header PH, the ROIinformation, and the compressed image data input from the encodingsection 125 via the ROI data buffer 145. The combining section 147outputs the generated transmission data 147A to the video receivingapparatus 200 via the data lane DL. Specifically, the combining section147 includes the respective data types of the regions of interest ROI1through ROI4 in the packet header PH of the payload data of a longpacket and sends out the data. Furthermore, the combining section 147sends out the respective image data (compressed image data) of theregions of interest ROI1 through ROI4 in a common virtual channel.

In a case where objects as imaging targets are not of a rectangularshape, the positional information of the ROI images 112 a 1 through 112a 4 is included in the packet header PH or the payload data of a longpacket. The positional information of the ROI images 112 a 1 through 112a 4 is included in the packet header PH by the PH generating section143. On the other hand, the positional information of the ROI images 112a 1 through 112 a 4 is included in the payload data of a long packet bythe combining section 147.

FIG. 12 is a diagram illustrating a configurational example of thetransmission data 147A where the positional information of the ROIimages 112 a 1 through 112 a 4 is included in the payload data of a longpacket. As illustrated in FIG. 12, the transmission data 147A include aframe header area R1 and a packet area R2, for example. Incidentally,FIG. 12 illustrates details of the contents of the frame header area R1.Furthermore, low power modes LP are omitted from illustration in FIG.12.

The frame header area R1 includes a frame number F1 as an identifier ofthe transmission data 147A, for example. The frame header area R1includes information regarding compressed image data included in thepacket area R2. The frame header area R1 includes, for example, thenumber of compressed image data (the number of ROIs) and information(ROI information) regarding each of the ROI images 112 a 1 through 112 a4 corresponding to each compressed image data. The ROI informationincludes region numbers, physical region lengths, rectangular outputregion sizes, priority, exposure information, gain information, AD wordlengths, and image formats. A physical region length represents themaximum length of a ROI image, and a rectangular output region sizerepresents the size of a region of interest ROI.

“Info” illustrated in FIG. 12 represents region information stored inthe payload of a long packet. The positional information of the ROIimages 112 a 1 through 112 a 4 is stored in “info,” for example. Thepositional information of the ROI images 112 a 1 through 112 a 4 isstored in the leading portions of the payloads of long packets. In acase where the physical region lengths in the X-axis direction ofsuccessive pixel rows making up ROI images are the same and each pixelrow does not include a ROI image of a different region number, theregion information “info” may not be stored in the payloads of longpackets including image data of second and following ones of the pixelrows. According to the present example, in the ROI image 112 a 1, thephysical region lengths in the X-axis direction of successive firstthrough fourth ones of all the pixel rows are the same, and the firstthrough fourth pixel rows do not include a ROI image of a differentregion number. Therefore, the region information “info” is not stored inthe payloads of respective long packets including the image data of thesecond through fourth pixel rows that correspond to second and followingones of the successive first through fourth pixel rows making up the ROIimage 112 a 1. According to the present example, furthermore, in the ROIimage 112 a 4, the physical region lengths in the X-axis direction ofsuccessive second and third ones of all the pixel rows are the same, andthe second and third pixel rows do not include a ROI image of adifferent region number. Therefore, the region information “info” is notstored in the payload of a long packet including the image data of thethird pixel row that corresponds to second and following ones of thesuccessive second and third pixel rows making up the ROI image 112 a 4.It is noted that, even in a case where the physical region lengths inthe X-axis direction are the same and the respective pixel rows do notinclude a ROI image of a different region number, the region information“info” may be stored in the payload of each row.

The combining section 147 divides and places compressed image datagenerated by compressing the respective ROI images 112 a 1 through 112 a4 per pixel row in the packet area R2 of the transmission data 147A, forexample. “1” illustrated in FIG. 12 represents the compressed image dataof the ROI image 112 a 1 stored in the payloads of long packets. “2”illustrated in FIG. 12 represents the compressed image data of the ROIimage 112 a 2 stored in the payloads of long packets. “3” illustrated inFIG. 12 represents the compressed image data of the ROI image 112 a 3stored in the payloads of long packets. “4” illustrated in FIG. 12represents the compressed image data of the ROI image 112 a 4 stored inthe payloads of long packets. In FIG. 12, the compressed image data areillustrated as being divided for an easy understanding. However, thedata stored in the payloads of long packets are not divided. Compressedimage data 112 b corresponding to the image of the region of overlap ROOare not overlappingly included in the packet area R2 of the transmissiondata 147A. Furthermore, the combining section 147 has eliminated pixelrows that do not correspond to respective transmission images of thecaptured image 111 from the packet area R2 of the transmission data147A. Consequently, pixel rows that do not correspond to respectivetransmission images of the captured image 111 are not included in thepacket area R2 of the transmission data 147A.

Next, operation of the video receiving apparatus 200 in a case where ithas received transmission data 147A will be described below.

The header separating section 211 of the receiving section 210 (see FIG.8 for both) receives transmission data 147A from the video transmittingapparatus 100 via the data lane DL. Specifically, the header separatingsection 211 receives transmission data 147A including ROI informationregarding the regions of interest ROI1 through ROI4 in the capturedimage 111 in the embedded data and also including image data (compressedimage data) of the regions of interest ROI1 through ROI4 in the payloaddata of long packets. The header separating section 211 separates thereceived transmission data 147A into a frame header area R1 and a packetarea R2.

The header interpreting section 212 (see FIG. 8) specifies the positionsof the payload data of long packets included in the packet area R2 onthe basis of data (specifically, embedded data) included in the frameheader area R1.

The payload separating section 213 (see FIG. 8) separates the payloaddata of the long packets included in the packet area R2 from the packetarea R2 on the basis of the positions of the payload data of the longpackets that have been specified by the header interpreting section 212.

The EBD interpreting section 214 outputs the embedded data as EBD datato the information processing section 220 (see FIG. 8). Furthermore, theEBD interpreting section 214 discriminates whether the image dataincluded in the payload data of the long packets are the compressedimage data of the image data 116 of a ROI or the compressed image dataof normal image data, from the data type included in the embedded data.The EBD interpreting section 214 outputs the discriminated result to theROI data separating section 215 (see FIG. 8).

If image data where the image data included in the payload data of longpackets represent a ROI are input, then the ROI data separating section215 outputs the payload data of the long packets as payload data to theinformation processing section 220 (specifically, the ROI decodingsection 222). The payload data of the long packets including ROIinformation include the ROI information and one line of pixel data ofthe compressed image data.

The information extracting section 221 (see FIG. 8) provided in theinformation processing section 220 extracts the number (four in thepresent example) of the regions of interest ROI1 through ROI4 includedin the captured image 111, the region numbers 1 through 4 and thepriorities 1 through 4 of the regions of interest ROI1 through ROI4, thedata lengths of the respective regions of interest ROI1 through ROI4,and the image formats of the respective regions of interest ROI1 throughROI4 from the embedded data included in the EBD data input from the EBDinterpreting section 214. Furthermore, the information extractingsection 221 extracts the positional information of the ROI images 112 a1 through 112 a 4 from the embedded data.

The ROI decoding section 222 decodes compressed image data 147B includedin the payload data to extract the positional information of the ROIimages 112 a 1 through 112 a 4 and generate image data (making uptransmission images). In a case where payload data corresponding to asixth pixel row, for example, are input, the ROI decoding section 222extracts one piece of positional information of the ROI image 112 a 1and two pieces of positional information of the ROI image 112 a 2 fromthe payload data, and generates respective image data (transmissionimages) of the ROI images 112 a 1 and 112 b 1 corresponding to the sixthpixel row.

In a case where payload data corresponding to a tenth pixel row, forexample, are input, the ROI decoding section 222 extracts one piece ofpositional information of the ROI image 112 a 3 and one piece ofpositional information of the ROI image 112 a 4 from the payload data,and generates respective image data (transmission images) of the ROIimages 112 a 3 and 112 b 4.

The ROI image generating section 223 (see FIG. 8) generates ROI images112 a 1 through 112 a 4 of the regions of interest ROI1 through ROI4 inthe captured image on the basis of the ROI information obtained by theinformation extracting section 221, the positional information of theROI images 112 a 1 through 112 a 4 extracted by the ROI decoding section222, and the transmission images generated by the ROI decoding section222. In a case where the one piece of positional information of the ROIimage 112 a 1 and two pieces of positional information of the ROI image112 a 2, extracted from the payload data, corresponding to the sixthpixel row, for example, and their transmission images are input, the ROIimage generating section 223 generates a ROI image 112 a 1 of fivepixels extending in the X-axis direction, a ROI image 112 a 2 of fourpixels extending in the X-axis direction at a position spaced fivepixels from the ROI image 112 a 1, and a ROI image 112 a 2 of two pixelsextending in the X-axis direction at a position spaced two pixels fromthe ROI image 112 a 2 (see FIG. 10).

Furthermore, the ROI image generating section 223 detects a region ofoverlap ROO where the region of interest ROI3 and the region of interestROI4 overlap each other on the basis of the ROI information obtained bythe information extracting section 221. The ROI image generating section223 generates a ROI image 112 a 3 of four pixels extending in the X-axisdirection and a ROI image 112 a 4 of three pixels extending in theX-axis direction with one pixel overlapping the ROI image 112 a 3 on thebasis of the detected region of overlap ROO, the respective positionalinformation of the ROI images 112 a 3 and 112 a 4, extracted from thepayload, corresponding to the tenth pixel row, and the transmissionimages (see FIG. 10).

The ROI image generating section 223 outputs the generated images as ROIimages to an apparatus at a subsequent stage (not illustrated).

In such a manner, the video transmitting apparatus 100 and the videoreceiving apparatus 200 can send and receive images of objects asimaging targets as ROI images even if the objects are of a shape otherthan a rectangular shape.

3. Principles of a Correcting Process for Correcting a Defective Pixelin an Embodiment of the Present Disclosure

Next, principles of a correcting process for correcting a defectivepixel in an embodiment of the present disclosure will be described belowwith reference to FIGS. 13 and 14.

FIG. 13 is a diagram schematically illustrating a correcting process forcorrecting a defective pixel according to the present embodiment. It isassumed that an image capturing region provided in an image capturingdevice includes a red pixel (hereinafter referred to as “R pixel”)disposed at a left upper end and includes odd-numbered rows where an Rpixel is located at a left end and R pixels and green pixels(hereinafter referred to as “G pixel”) are alternately disposed andeven-numbered rows where a G pixel is located at a left end and G pixelsand blue pixels (hereinafter referred to as “B pixel”) are alternatelydisposed. FIG. 13 schematically illustrates a portion of an imagecaptured by the image capturing device that has such a pixel structure.

The image illustrated in FIG. 13 is an unprocessed image, called Rawimage or undeveloped data, output from the pixels to the image capturingregion. If a photoelectric transducer or a pixel circuit suffers afailure, resulting in a pixel defect that causes a pixel to malfunction,the pixel acquires an image as a defective image Id where no desiredgradation is obtained, as indicated on the left side of the thick arrowin FIG. 13.

The Raw image that includes the defective image Id tends to raise apossibility that an image to be generated finally therefrom may have itsimage quality degraded. According to the present embodiment, asindicated on the right side of the thick arrow in FIG. 13, theinformation of the defective image Id is interpolated using theinformation of peripheral images Iad that are identical in color to thedefective image Id and disposed around the defective image Id,generating a corrective image Ic. In such a manner, according to thepresent embodiment, the defect of the image is rendered less noticeable,preventing an image to be generated finally therefrom from having adegraded image quality.

Defective pixels are produced when image capturing devices aremanufactured and assembled and result in fixed defects in the imagecapturing devices. Consequently, it is possible to detect defectivepixels in a delivery inspection or the like of image capturing devices,store coordinate information of the defective pixels in a nonvolatilestorage device such as an EEPROM, and correct defective images using thecoordinate information.

In a case where an image of an overall image capturing area isgenerated, it is possible to correct a defective image according to sucha process. However, regions of interest ROI that are segmented haveindefinite scopes and sizes. Therefore, as the position of a defectivepixel in a segmented region of interest cannot be predicted, it isimpossible to correct a defective image in the region of interest usingthe stored coordinate information of defective pixels.

Accordingly, in the transmitting apparatus, the receiving apparatus, andthe transmission system according to the present embodiment, the videotransmitting apparatus that has the image capturing section isconfigured to calculate the position of a defective pixel in a region ofinterest on the basis of coordinate information of the defective pixelthat is acquired in a delivery inspection or the like. Furthermore, thevideo receiving apparatus is configured to perform a correcting processfor correcting a defective image using the position of the defectivepixel that is sent out from the video transmitting apparatus.

FIG. 14 is a diagram schematically illustrating a method of calculatingthe position of a defective pixel in a region of interest on the basisof coordinate information of the defective pixel acquired in a deliveryinspection or the like according to the present embodiment.

As illustrated in FIG. 14, an image capturing region IR provided by aplurality of image capturing elements has six defective pixels D01, D02,D11, D12, D21, and D31 present therein, for example. In addition, threeregions of interest ROI1 through ROI3 to be segmented are established inthe image capturing region IR. The defective pixels D11 and D12 aredisposed in a region where the region of interest ROI1 is established.The defective pixel D21 is disposed in a region where the region ofinterest ROI2 is established. The defective pixel D31 is disposed in aregion where the region of interest ROI3 is established. The defectivepixels D01 and D02 are established in a region where no regions ofinterest are established. The method of calculating the position of adefective pixel in a region of interest is common regardless of thepositions of regions of interest and defective pixels. Therefore, themethod of calculating the position of a defective pixel in a region ofinterest will be described below with respect to the defective pixelD11, for example.

As illustrated in FIG. 14, the image capturing region IR has its originIRo (0, 0) located on a pixel at a left upper end thereof. In this case,the defective pixel D11 in the image capturing region IR has coordinates(x, y) represented respectively by (D1_X1, D1_Y1). The coordinates(D1_X1, D1_Y1) of the defective pixel D11 are coordinates based on theorigin IRo (0, 0) of the image capturing region IR. Furthermore, thecoordinates (D1_X1, D1_Yl) of the defective pixel D1 l have informationas coordinate information acquired upon a delivery inspection, forexample, of the video transmitting apparatus that has the imagecapturing section.

In a case where the regions of interest are established as describedabove, positional information of the region of interests (for example,the coordinates of a left upper end thereof, the length in the X-axisdirection thereof, and the length in the Y-axis direction thereof) isderived. Consequently, as illustrated in FIG. 14, the coordinates (R1_X,R1Y) of a left upper end Pr, the length R1_W in the X-axis direction,and the length R1_H in the Y-axis direction, for example, are derived aspositional information of the region of interest ROI1. The coordinates(R1_X, R1Y) of the left upper end Pr of the region of interest ROI1represent the position of the region of interest ROI1 in the imagecapturing region IR. Therefore, the coordinates (R1_X, R1Y) of the leftupper end Pr are coordinates based on the origin IRo (0, 0) of the imagecapturing region IR.

Accordingly, the coordinates (R1_X, R1Y) of the left upper end Pr of theregion of interest ROI1 and the coordinates (D1_X1, D1_Yl) of thedefective pixel D11 are coordinates defined in the same coordinate spaceas the image capturing region IR. Therefore, the coordinates (D1X_R1,D1_YR1) of the defective pixel D11 in the region of interest ROI1 can beexpressed by the following equations (1) and (2):

D1_XR1=D1_X−R1_X  1)

D1_YR1=D1_Y−R1_Y  2)

Because of the equations (1) and (2), the position of the defectivepixel D11 can be expressed by coordinates in the coordinate space of theregion of interest ROI1. Therefore, by storing coordinate informationacquired upon a delivery inspection oz the like, for example, the videoreceiving apparatus can calculate coordinates of a defective pixel in aregion of interest each time the region of interest is established andpositional information (the coordinates of a left upper end, the lengthin the X-axis direction, and the length in the Y-axis direction) of theregion of interest is input from the video transmitting apparatus.

As with the video transmitting apparatus 100 described above, thetransmitting apparatus according to the present embodiment is configuredto send out in normal operation ROI information including positionalinformation (the coordinates of a left upper end, the length in theX-axis direction, and the length in the Y-axis direction) of a region ofinterest as embedded data to the video receiving apparatus. Furthermore,as described in detail later, the transmitting apparatus according tothe present embodiment is configured to send out in an initializingprocess defect correcting information including information ofcoordinates of a defective pixel in the coordinate space of an imagecapturing region (i.e., a full angle of view) as embedded data to thevideo receiving apparatus.

As with the video receiving apparatus 200 described above, the receivingapparatus according to the present embodiment can extract informationincluded in embedded data. Therefore, the receiving apparatus accordingto the present embodiment can calculate coordinates of a defective pixelin a region of interest according to the equations (1) and (2) using thecoordinates of the defective pixel extracted from embedded data receivedin an initializing process and coordinates of a left upper end of theregion of interest received in normal operation. Furthermore, thereceiving apparatus is configured to use the coordinates of thedefective pixel acquired by the calculation as the coordinates of adefective image.

By subtracting 2 from the value of the Y coordinate determined accordingto the equation (2) while keeping the value of the X coordinatedetermined according to the equation (1) as it is, coordinates of apixel that is two rows ahead of the defective pixel D11 are determined.Furthermore, by adding 2 to the value of the Y coordinate determinedaccording to the equation (2) while keeping the value of the Xcoordinate determined according to the equation (1) as it is,coordinates of a pixel that is two rows behind the defective pixel Dl1are determined. Moreover, by subtracting 2 from the value of the Xcoordinate determined according to the equation (1) while keeping thevalue of the Y coordinate determined according to the equation (2) as itis, coordinates of a pixel that is two columns ahead of the defectivepixel D11 are determined. Furthermore, by adding 2 to the value of the Xcoordinate determined according to the equation (1) while keeping thevalue of the Y coordinate determined according to the equation (2) as itis, coordinates of a pixel that is two rows behind the defective pixelD11 are determined.

Images corresponding to the pixels thus determined correspond toperipheral images (see the peripheral images Iad illustrated in FIG. 13)around a defective image corresponding to the defective pixel D11.Therefore, the receiving apparatus according to the present embodimentcan also calculate coordinates of peripheral images using thecoordinates of a defective pixel and the coordinates of a left upper endof a region of interest that are included in the embedded data. Thereceiving apparatus according to the present embodiment can thus performa correcting process for correcting a defective image that is present ina region of interest regardless of the position and size of the regionof interest. According to the present embodiment, since the Raw image isof a Bayer array, images of pixels having coordinates that are t2 withrespect to the coordinates of the defective pixel are used as peripheralimages. However, peripheral images may be selected appropriatelydepending on the array of color pixels.

4. A First Embodiment of the Present Disclosure

Next, a transmitting apparatus, a receiving apparatus, and atransmission system according to a first embodiment of the presentdisclosure will be described below with reference to FIGS. 15 through22. First, a general makeup of the transmitting apparatus, the receivingapparatus, and the transmission system according to the presentembodiment will be described below with reference to FIG. 15. FIG. 15 isa block diagram illustrating a general makeup of a video transmittingapparatus 3, a video receiving apparatus 4, and a video transmissionsystem 10 according to the present embodiment.

As illustrated in FIG. 15, the video transmission system 10 according tothe present embodiment includes the video transmitting apparatus (anexample of the transmitting apparatus) 3 that functions as an imagesensor and the video receiving apparatus (an example of the receivingapparatus) 4 that functions as an image signal processor (ISP). In thevideo transmission system 10, the video transmitting apparatus 3 isconfigured to have a transmitting section 322 send out signals accordingto the MIPI (Mobile Industry Processor Interface) D-PHY standards, theMIPI C-PHY standards, or the MIPI CSI (Camera Serial Interface)-2standards. In the video transmission system (an example of transmissionsystem) 10, furthermore, the video receiving apparatus 4 is configuredto have a receiving section 412 receive signals according to the MIPID-PHY standards, the MIPI C-PHY standards, or the MIPI CSI-2 standards.Moreover, the video transmission system 10 may be configured to send andreceive signals according to the MPIP CSI-3 standards or the MIPI DSIstandards between the video transmitting apparatus 3 and the videoreceiving apparatus 4, as with the video transmission system 1 accordingto the presupposed technologies 1 and 2.

The video transmitting apparatus 3 provided in the video transmissionsystem 10 is configured to perform functions equivalent to those of thevideo transmitting apparatus 100 according to the presupposedtechnologies 1 and 2. Specifically, the video transmitting apparatus 3is configured to perform the same process as the video transmittingapparatus 100 on captured images input from an image capturing section31 in a case where a control signal indicating the segmentation of ROIsis input from the video receiving apparatus 4. Furthermore, the videotransmitting apparatus 3 is configured to perform the same process asthe video transmitting apparatus 100 on captured images input from theimage capturing section 31 in a case where a control signal indicatingthe outputting of a normal image is input from the video receivingapparatus 4.

Furthermore, the video transmitting apparatus 3 is configured totransmit in an initializing process all information (an example ofdefect correcting information) of the coordinates of a defective pixelamong all pixels (a fully angle of view) of an image capturing region ofthe image capturing section 31 (to be described in detail later) in acase where a control signal requesting the acquisition of thecoordinates of the defective pixel is input from the video receivingapparatus 4.

The video receiving apparatus 4 is configured to perform functionsequivalent to those of the video receiving apparatus 200 according tothe presupposed technologies 1 and 2. Specifically, the video receivingapparatus 4 is configured to perform a similar process to the videoreceiving apparatus 200 according to the presupposed technologies 1 and2 on transmission data transmitted from the video transmitting apparatus3. Furthermore, the video receiving apparatus 4 is configured to performa correcting process for correcting a defective image based on adefective pixel using defect correcting information transmitted from thevideo transmitting apparatus 3.

Therefore, FIG. 15 illustrates the video transmitting apparatus 3 andthe video receiving apparatus 4 mainly with respect to configurationaldetails regarding the correcting process for correcting a defectivepixel.

As illustrated in FIG. 15, the video transmitting apparatus 3 includesthe image capturing section 31 that captures images of targets. Theimage capturing section 31 has a photoelectric converting section 311for converting incident light into electric signals, for example. Thephotoelectric converting section 311 includes, for example, a CCD imagesensor or a CMOS image sensor. The photoelectric converting section 311has a plurality of pixels each having a photoelectric transducer andarrayed according to predetermined rules. The photoelectric convertingsection 311 includes an image capturing region where the pixels arearrayed. Furthermore, the image capturing section 31 has a signalconverting section 312 for converting an analog electric signal inputfrom the photoelectric converting section 311 into digital image data.The signal converting section 312 is configured to perform a signalamplifying (AGC) process for amplifying the analog electric signal inputfrom the photoelectric converting section 311 and an analog-to-digitalconverting (ADC) process for converting the amplified signal into adigital signal. The image capturing section 31 has an amplifying section313 for applying a digital gain to image data input from the signalconverting section 312. The amplifying section 313 outputs the imagedata with the digital gain applied thereto to the transmitting section322.

The video transmitting apparatus 3 has a nonvolatile storage device 33.The nonvolatile storage device 33 includes an EEPROM (ElectricallyErasable Programmable Read-Only Memory), for example. The nonvolatilestorage device 33 stores various pieces of information, e.g., thecoordinates of all defective pixels among the pixels of thephotoelectric converting section 311. The coordinates of the defectivepixels are detected upon a delivery inspection, for example, of thevideo transmitting apparatus 3 and are stored.

The video transmitting apparatus 3 includes a controlling section 32 forcontrolling the image capturing section 31 and controlling predeterminedsignal processing processes. The controlling section 32 has a sensor CPU321 and the transmitting section 322. The sensor CPU 321 is configuredto perform similar functions to the image processing sections 120 and130 (see FIG. 2). The transmitting section 320 is configured to performsimilar functions to the transmitting section 140 (see FIG. 2). In thecontrolling section 32, the sensor CPU 321 may be replaced with imageprocessing sections 120 and 130, and the transmitting section 322 may bereplaced with the transmitting section 140.

The sensor CPU 321 has an exposure controlling section 321 a forcontrolling exposure conditions of the photoelectric converting section311. Furthermore, the sensor CPU 321 has a conversion area controllingsection (an example of the controlling section) 321 b for controllingthe acquisition of defect correcting information as information forcorrecting a defect of an image included in a region of interest ROI.Each of the sensor CPU 321 having the conversion area controllingsection 321 b and the controlling section 32 corresponds to an exampleof a controlling section for controlling the acquisition of defectcorrecting information as information for correcting a defect of animage included in a region of interest ROI.

The conversion area controlling section 321 b is configured to acquire,as defect correcting information, defect coordinates representingcoordinates of a pixel where a defect has occurred among the pixelsmaking up the image capturing region of the photoelectric convertingsection 311. The conversion area controlling section 321 b is configuredto acquire information of the defect coordinates at the time the videotransmitting apparatus 3 is initialized or manufactured (e.g., upon adelivery inspection) from the nonvolatile storage device 33 and hold theacquired information, for example.

The sensor CPU 321 transmits, in an initializing process, for example,information of defect coordinates in the image capturing region (i.e., afull angle of view) of a defective pixel acquired by the conversion areacontrolling section 321 b to an Isp CPU 411 (to be described in detaillater) in the video receiving apparatus 4 via inter-CPU communication.

In a case where a control signal indicating the segmentation of ROIs isinput from the video receiving apparatus 200 via the camera controlinterface CCI in a process of normal operation, the sensor CPU 321specifies an object as an imaging target included in the captured image.Furthermore, the sensor CPU 321 sets a region of interest ROI for theidentified object, assigns a region number to the region of interestROI, segments an image of the region of interest ROI, and stores theimage in a storage section (not illustrated) in association with theregion number. Moreover, the sensor CPU 321 (specifically, theconversion area controlling section 321 b) calculates positionalinformation (for example, the coordinates of a left upper end (anexample of reference coordinates), the length in the X-axis direction,and the length in the Y-axis direction) of the region of interest ROI,and stores the positional information in the storage section inassociation with the region number of the region of interest ROI.Specifically, the conversion area controlling section 321 b isconfigured to acquire information (i.e., positional information of theregion of interest ROI) of the coordinates of the left upper end of theregion of interest ROI (an example of basic coordinates) and the lengthin the X-axis direction and the length in the Y-axis direction (anexample of the size of the region of interest) of the region of interestROI, as defect correcting information, and to hold the acquiredinformation upon normal operation of the video transmitting apparatus 3.In this case, the conversion area controlling section 321 b acquires andholds the coordinates of the left upper end as basic coordinates.

The sensor CPU 321 outputs the acquired positional information of theregion of interest ROI, image data input from the image capturingsection 31, etc. to the transmitting section 322. The transmittingsection 322 generates transmission data (see FIGS. 6 and 12) includingthese pieces of information input from the sensor CPU 321 and sends outthe generated transmission data to the video receiving apparatus 4.

As illustrated in FIG. 15, the video transmitting apparatus 3 includesthe transmitting section 322 that sends out the image data of an imageincluded in the region of interest ROI as the payload data of a longpacket and sends out ROI information as the embedded data. The videotransmitting apparatus 3 sends out information (i.e., positionalinformation of the region of interest ROI) of the basic coordinates andsize of the region of interest ROI as one piece of ROI information, asdefect correcting information from the transmitting section 322 uponnormal operation of the video transmission system 10. In this case, thevideo transmitting apparatus 3 sends out the coordinates of the leftupper end of the region of interest ROI as reference coordinates fromthe transmitting section 322. Furthermore, the video transmittingapparatus 3 is configured to transmit the coordinates of a pixel where adefect has occurred among the pixels making up the image capturingregion of the photoelectric converting section 311, as defect correctinginformation from the transmitting section 322 at the time the videotransmission system 10 is initialized or manufactured (e.g., upon adelivery inspection), for example. The video transmitting apparatus 3sends out the defect correcting information included in the ROIinformation from the transmitting section 322. The transmitting section322 is configured to send out transmission data including demosaicinginformation according to the MIPI D-PHY standards, the MIPI C-PHYstandards, or the MIPI CSI-2 standards.

As illustrated in FIG. 15, the video receiving apparatus 4 includes acontrolling section 41 for controlling a predetermined signal processingprocess using transmission data transmitted from the video transmittingapparatus 3. The controlling section 41 has an Isp CPU 411, a receivingsection 412, and an embedded data acquiring section 413. The Isp CPU 411is configured to perform similar functions to the information processingsection 220 (see FIG. 8), except for the information extracting section221 and the ROI image generating section 223 (see FIG. 8). In the videoreceiving apparatus 4, an image generating section 422 is configured toperform similar functions to the ROI image generating section 223. Thereceiving section 412 is configured to perform similar functions to thereceiving section 210 (see FIG. 8), except for the EBD interpretingsection 214 (see FIG. 8). In the video receiving apparatus 4, theembedded data acquiring section 413 is configured to perform similarfunctions to the EBD interpreting section 214 and the informationextracting section 221. In the controlling section 41, the receivingsection 412 and the embedded data acquiring section 413 may be replacedwith the receiving section 210, and the Isp CPU 411 and the imagegenerating section 422 may be replaced with the information processingsection 220. In this case, the functions of the information extractingsection 221 that are performed by the embedded data acquiring section413 are performed by the receiving section 220.

As illustrated in FIG. 15, the video receiving apparatus 4 includes thereceiving section 412 that receives a transmission signal where imagedata of images included in regions of interest ROI are included in thepayload data and ROI information is included in the embedded data. Thereceiving section 412 is configured to receive transmission data inputfrom the video transmitting apparatus 3. The transmission data inputfrom the video transmitting apparatus 3 include ROI information havingdefect correcting information. The ROI information is included in theembedded data. Therefore, the transmission data input from the videotransmitting apparatus 3 includes the embedded data that include thedefect correcting information. The ROI information that is received bythe video receiving apparatus 4 with the receiving section 412 when thevideo transmission system 10 is in normal operation includes informationof the reference coordinates and sizes of the regions of interest ROI asthe defect correcting information. Therefore, the video receivingapparatus 4 receives, with the receiving section 412, the referencecoordinates and sizes of the regions of interest ROI as the defectcorrecting information. The video receiving apparatus 4 receives, withthe receiving section 412, the coordinates of the left upper ends of theregions of interest ROI as the reference coordinates. Furthermore, thevideo receiving apparatus 4 receives, with the receiving section 412,pixels where defects have occurred in a plurality of pixels making upthe image capturing region provided in the photoelectric convertingsection 311 at the time the video transmission system 10 is initializedor manufactured (e.g., upon a delivery inspection), for example. Thereceiving section 412 receives the transmission data according to theMIPI D-PHY standards, the MIPI C-PHY standards, or the MIPI CSI-2standards. Therefore, the video receiving apparatus 4 is configured toreceive, with the receiving section 412, signals according to the MIPID-PHY standards, the MIPI C-PHY standards, or the MIPI CSI-2 standards.The receiving section 412 generates various data from the inputtransmission data and outputs the generated data to the Isp CPU 411 andthe embedded data acquiring section 413.

As illustrated in FIG. 15, the video receiving apparatus 4 includes theembedded data acquiring section (an example of the controlling section)413 that extracts defect correcting information as information used forcorrecting defects of image data of images included in the regions ofinterest ROI from the transmission data (an example of a transmissionsignal) received by the receiving section 412. The controlling section41 having the embedded data acquiring section 413 corresponds to anexample of a controlling section for controlling the extraction ofdefect correcting information used as information for correcting defectsof image data of images included in the regions of interest ROI from atransmission signal (transmission data) received by the receivingsection 412. The embedded data acquiring section 413 is configured toextract the defect correcting information from the ROI informationincluded in the transmission data input from the receiving section 412.Since the ROI information is included in the embedded data, the embeddeddata acquiring section 413 extracts the defect correcting informationfrom the embedded data included in the transmission signal (transmissiondata) input from the receiving section 412. Consequently, the videoreceiving apparatus 4 is configured to receive, with the receivingsection 412, the transmission data including the embedded data havingthe defect correcting information, and to extract, with the embeddeddata acquiring section 413, the defect correcting information from theROI information included in the embedded data included in thetransmission data received by the receiving section 412.

The embedded data acquiring section 413 is configured to acquire thereference coordinates and sizes of the regions of interest ROI as thedefect correcting information from the ROI information included in theembedded data when the video receiving apparatus 4 is in normaloperation. The embedded data acquiring section 413 acquires thecoordinates of the left upper ends of the regions of interest ROI as thereference coordinates of the regions of interest ROI. Furthermore, theembedded data acquiring section 413 acquires the lengths in the X-axisdirection and the lengths in the Y-direction of the regions of interestROI as the sizes of the regions of interest ROI. In such a manner, theembedded data acquiring section 413 acquires the positional informationof the regions of interest ROI from the embedded data as the defectcorrecting information when the video receiving apparatus 4 is in normaloperation, and outputs the defect correcting information acquired fromthe embedded data to the Isp CPU 411.

The embedded data acquiring section 413 acquires, other than thedemosaicing information, various pieces of information (for example, thenumber of regions of interest ROI, the region numbers and priority ofthe regions of interest ROI, the data lengths of the regions of interestROI, the image format of the regions of interest ROI, etc.) included inthe embedded data. The embedded data acquiring section 413 outputs theacquired various pieces of information to the Isp CPU 411.

As illustrated in FIG. 15, the Isp CPU 411 has a coordinate convertingsection (an example of the controlling section) 411 a. The coordinateconverting section 411 a is configured to convert defect coordinates inthe image capturing region into defect coordinates in a region ofinterest ROI. When the positional information of a region of interestROI is input from the embedded data acquiring section 413 at the timethe video transmission system 10 is in normal operation, the coordinateconverting section 411 a converts defect coordinates in the imagecapturing region acquired when the video transmission system 10 has beeninitialized into corrective coordinates that are defect coordinates inthe region of interest ROI, using the positional information.Specifically, the coordinate converting section 411 a extracts thecoordinates of a pixel where a defect has occurred as coordinates in theimage capturing region (coordinates in an XY orthogonal coordinatesystem in the image capturing region) from a given storage region, forexample, and converts the extracted defect coordinates into defectcoordinates in the coordinate system of the region of interest ROI. Thecoordinate system of the region of interest ROI herein means an XYorthogonal coordinate system having its origin located on a pixel at aleft upper end of the image capturing region as illustrated in FIG. 14,for example, and extending in the X-axis direction (horizontaldirection) and the Y-axis direction (vertical direction) of the imagecapturing region. In addition, the coordinate system of the region ofinterest means an XY orthogonal coordinate system having its origin on areference pixel (pixel at the left upper end according to the presentembodiment) of the region of interest and extending in the X-axisdirection (horizontal direction) and the Y-axis direction (verticaldirection) of the region of interest.

The coordinate converting section 411 a specifies a range that a regionof interest ROI input from the embedded data acquiring section 413 takesup in the image capturing region provided in the photoelectricconverting section 311. For example, the coordinate converting section411 a specifies which position the region of interest ROI is disposed inin the coordinate system of the image capturing region, on the basis ofthe positional information (the coordinates of the left upper end, thelength in the X-axis direction, and the length in the Y-axis direction)of the region of interest ROI. Next, the coordinate converting section411 a converts defect coordinates present in the region of interest ROIin the image capturing region into coordinates in the coordinate systemof the region of interest ROI, thereby acquiring correction targetcoordinates. The coordinate converting section 411 a carries out acoordinate conversion on defect coordinates according to the aboveequations (1) and (2), thereby acquiring correction target coordinates.In a case where positional information of a plurality of regions ofinterest ROI is input, the coordinate converting section 411 a carriesout a coordinate conversion on defect coordinates in each of the regionsof interest ROI. In such a manner, the coordinate converting section 411a determines a position in the region of interest ROI that correspondsto the coordinates of a pixel where a defect has occurred (according tothe present embodiment, the coordinates in the XY orthogonal coordinatesystem of the region of interest ROI) on the basis of the referencecoordinates of the region of interest ROI (for example, the coordinatesof the left upper end and the size (for example, the length in theX-axis direction, and the length in the Y-axis direction) thereof andthe coordinates of the pixel where the defect has occurred (according tothe present embodiment, the coordinates in the XY orthogonal coordinatesystem of the image capturing region).

The video receiving apparatus 4 may store the defect correctinginformation (i.e., defect coordinates) acquired at the time the videotransmission system 10 is initialized in a given storage region in thecontrolling section 41 or the Isp CPU 411, and may store the defectcorrecting information in a separate storage section that the videoreceiving apparatus 4 has.

The Isp CPU 411 outputs information of the defect coordinates convertedby the coordinate converting section 411 a to an image processingsection 42 in association with the region number (indicated as “ROI ID”in FIG. 15) of the region of interest ROI.

As illustrated in FIG. 15, the video receiving apparatus 4 includes theimage processing section 42. The image processing section 42 has astatic defect correcting section (an example of the processing section)421 for processing a correction on a defect in the image of the regionof interest ROI on the basis of the defect correcting informationextracted by the embedded data acquiring section 413. The imageprocessing section 42 that has the static defect correcting section 421corresponds to an example of the processing section for processing acorrection on a defect in the image of the region of interest ROI on thebasis of the defect correcting information extracted by the embeddeddata acquiring section 413. Furthermore, the image processing section 42has an image generating section 422 for performing Raw processing andRGB processing on the image data of the region of interest ROI includingan image where the image defect has been corrected. Moreover, the imageprocessing section 42 has an image quality adjusting section 422 foradjusting the image quality of the image data that have been processedby the image generating section 422.

More specifically, the static defect correcting section 421 is arrangedto correct a defect in the image of the region of interest ROI using thecorrection target coordinates acquired by the coordinate convertingsection 411 a. The correction target coordinates are acquired in theform of the defect coordinates included in the embedded data andtransmitted from the video transmitting apparatus 3 to the videoreceiving apparatus 4 as the defect correcting information and thepositional information of the region of interest ROI. Therefore, thestatic defect correcting section 421 corrects a defect in the image ofthe region of interest ROI using the correction target coordinatesacquired from the defect correcting information, though not directlyusing the defect correcting information. Consequently, the static defectcorrecting section 421 corrects a defect in the image of the region ofinterest ROI on the basis of the defect correcting information extractedby the embedded data acquiring section 413.

The static defect correcting section 421 determines coordinates ofperipheral images to be used for correcting a defective image on thebasis of the correction target coordinates. The static defect correctingsection 421 has a line memory for several rows (e.g., five rows) of theimage capturing region provided in the photoelectric converting section311. The static defect correcting section 421 stores the image data ofthe region of interest ROI input from the Isp CPU 411 in the linememory. Furthermore, in a case where a defective image and peripheralimages to be used for correcting the defective image are input, thestatic defect correcting section 421 stores the input images in storageareas having predetermined addresses of the line memory. In a case wherea defective image and four peripheral images are stored in the storageareas, the static defect correcting section 421 performs a correctingprocess for correcting defective pixels. In such a manner, the staticdefect correcting section 421 is configured to correct all defectivepixels.

The static defect correcting section 421 is configured, for example, tostore image data of one line of the region of interest ROI where adefective image is present as a correction target in the line memory andoutput the image data to the image generating section 422. When thestatic defect correcting section 421 outputs the image data to the imagegenerating section 422, the static defect correcting section 421 outputsthe image data of a corrective image rather than a defective image tothe image generating section 422.

When the image data are input from the static defect correcting section421, the image generating section 422 acquires information (ROIinformation) regarding the region of interest ROI including the imagedata from the Isp CPU 411 and generates an image of the region ofinterest ROI. For generating an image of the region of interest ROI, theimage generating section 422 first performs a Raw process to generate aregion of interest ROI constructed as a Raw image. Next, the imagegenerating section 422 performs an RGB process to generate an image ofthe region of interest ROI including image data of RGB signals. Theimage generating section 422 outputs the image data of RGB signalsmaking up the region of interest ROI to the image quality adjustingsection 423.

The image quality adjusting section 423 is configured to perform aninverse RGB process for converting the image data of RGB signals inputfrom the image generating section 422 into a luminance signal and twocolor difference signals. Furthermore, the image quality adjustingsection 423 is configured to perform a gamma correcting process on theimage data processed by the inverse RGB process. Moreover, the imagequality adjusting section 423 is configured to perform a YC process suchas color difference correction, noise reduction, etc. on image data ofthe luminance signal and the two color difference signals. The imagequality adjusting section 422 is configured to output an image adjustedin image quality to a display device (not illustrated), for example. Inthis fashion, a desired image where defective images have been correctedand image quality has been adjusted is displayed on the display image.

(Correcting Process for Correcting a Defective Pixel)

Next, a correcting process for correcting a defective pixel in thetransmitting apparatus, the receiving apparatus, and the transmissionsystem according to the present embodiment will be described below usingFIGS. 16 through 19 with reference to FIG. 15. First, a sequence of thecorrection of a defective pixel in the transmitting apparatus, thereceiving apparatus, and the transmission system according to thepresent embodiment will be described below with reference to FIGS. 16through 18. FIGS. 16 through 18 are flowcharts illustrating an exampleof the sequence of the correction of a defective pixel in thetransmitting apparatus, the receiving apparatus, and the transmissionsystem according to the present embodiment. First, an example of asequence of a process for acquiring coordinates of a defective pixel inan initializing process of the transmitting apparatus, the receivingapparatus, and the transmission system will be described below withreference to FIG. 16.

An initializing process of the video transmitting apparatus 3, the videoreceiving apparatus 4, and the video transmission system 10 according tothe present embodiment is carried out when the video transmittingapparatus 3, the video receiving apparatus 4, and the video transmissionsystem 10 are activated, i.e., when the video transmission system 10 isactivated.

(Step S11)

When the video transmission system 10 is activated, the Isp CPU 411 ofthe video receiving apparatus 4 sends out a control signal requestingthe video transmitting apparatus 3 to acquire the coordinates ofdefective pixels. Specifically, in step S11, the Isp CPU 411 requeststhe video transmitting apparatus 3 to acquire information of thecoordinates of all defective pixels in the image capturing region (i.e.,the full angle of view) provided in the photoelectric converting section311 of the image capturing section 31. After having the video receivingapparatus 4 request the acquisition of information of the coordinates ofthe defective pixels, the video transmission system 10 goes to theprocessing of step S13.

(Step S13)

In response to the control signal requesting the acquisition of thecoordinates of the defective pixels from the video receiving apparatus4, the video transmitting apparatus 3 reads the coordinates of thedefective pixels from the nonvolatile storage device 33. Specifically,the video transmitting apparatus 3 reads defect coordinates (i.e.,defect coordinates in the coordinate system of the image capturingregion) in the image capturing region (i.e., the full angle of view)provided in the photoelectric converting section 311 from thenonvolatile storage device 33. The video transmission system 10 sendsout the information of the read defect coordinates from the videotransmitting apparatus 3 to the video receiving apparatus 4 viainter-CPU communication between the sensor CPU 321 and the Isp CPU 411,and then goes to the processing of step S15.

(Step S15)

Having received the information of the defect coordinates output fromthe video transmitting apparatus 3, the video receiving apparatus 4stores the received information in a predetermined storage regionprovided in the controlling section 41 or the Isp CPU 411, for example.In a case where the video receiving apparatus 4 has acquired and savedthe information of the coordinates of the defective pixels, the videotransmission system 10 brings the initializing process to an end.

Next, a calculating process for the coordinate conversion of a defectivepixel upon normal operation of the video transmitting apparatus 3, thevideo receiving apparatus 4, and the video transmission system 10according to the present embodiment will be described below withreference to FIG. 17. FIG. 17 is a flowchart illustrating an example ofa sequence of a calculating process for the coordinate conversion of adefective pixel upon normal operation of the transmitting apparatus, thereceiving apparatus, and the transmission system.

(Step S31)

As illustrated in FIG. 17, when a frame starting trigger is input, thesensor CPU 321 provided in the video transmitting apparatus 3 determinesa segmenting position for segmenting an image from the image capturingregion of the image capturing section 31, and then goes to theprocessing of step S33. In step S31, the sensor CPU 321 determines thesegmenting position, i.e., coordinates of a left upper end of a regionof interest ROI, and an image size (lengths in the X-axis direction andthe Y-axis direction) thereof, and sets information of the determinedcoordinates and image size as defect correcting information in embeddeddata.

(Step S33)

The sensor CPU 321 sets transmission data including the embedded data inwhich the coordinates of the left upper end and the image size of theregion of interest ROI, the region number of the region of interest ROI,the image data included in the region of interest ROI, etc. in thetransmitting section 322, and brings the calculating process for thecoordinate conversion of a defective pixel to an end.

The transmission data set in step S33 is transmitted from the videotransmitting apparatus 3 to the video receiving apparatus 4 by way ofcommunication through hardware (HW) using MIPI.

The receiving section 412 included in the video receiving apparatus 4extracts the embedded data from the received transmission data andoutputs the embedded data to the embedded data acquiring section 413.The embedded data acquiring section 413 decodes the embedded data inputfrom the receiving section 412, acquires various pieces of information(for example, the number of regions of interest ROI, the region numbersand priority of the regions of interest ROI, the data lengths of theregions of interest ROI, the image format of the regions of interestROI, etc.), and outputs the acquired various pieces of information tothe Isp CPU 411.

(Step S41)

The Isp CPU 411, triggered by the timing at which the embedded data aredecoded by the embedded data acquiring section 413, acquires thesegmenting position and the size on the basis of the various pieces ofinformation acquired and input by the embedded data acquiring section413 from the transmission data received by the receiving section 412.Specifically, the Isp CPU 411 acquires positional information (thecoordinates of the reference pixel, the length in the X-axis direction,and the length in the Y-axis direction) of a region of interest ROIwhose priority is highest on the basis of the various pieces ofinformation acquired and input by the embedded data acquiring section413, and then goes to the processing of step S43. According to thepresent embodiment, the coordinates of the left upper end of the regionof interest ROI are acquired as the coordinates of the reference pixel.

(Step S43)

The Isp CPU 411 extracts a defective pixel that is present in the rangeof a segmented region, i.e., the region of interest ROI acquired in stepS41, and goes to the processing of step S45. Specifically, the Isp CPU411 reads the defect coordinates stored in the given storage region andextracts coordinates that agree with the coordinates in the range of theregion of interest ROI from the read defect coordinates.

(Step S45)

The Isp CPU 411 determines whether a defective pixel is present in therange of the region of interest ROI or not in step S43. If the Isp CPU411 determines that a defective pixel is present in the range of theregion of interest ROI (Yes), then the Isp CPU 411 goes to theprocessing of step S47. On the other hand, if the Isp CPU 411 determinesthat a defective pixel is not present in the range of the region ofinterest ROI (No), then the Isp CPU 411 goes to the processing of stepS49.

(Step S47)

The Isp CPU 411 (specifically, the coordinate converting section 411 a)carries out a coordinate calculation on the defective coordinatesextracted in step S43, and goes to the processing of step S49. In stepS47, the coordinate converting section 411 a converts the defectcoordinates extracted in step S43 (i.e., the defect coordinates in thecoordinate system of the image capturing region provided in thephotoelectric converting section 311) into coordinates (correctivecoordinates) in the coordinate system of the region of interest ROIacquired in step S41.

(Step S49)

The Isp CPU 411 determines whether the processing from step S41 to stepS47 has been performed on all the regions of interest ROI input from theembedded data acquiring section 413 or not. If the Isp CPU 411determines that the processing has been performed on all the regions ofinterest ROI (Yes), then the Isp CPU 411 ends the calculating processfor the coordinate conversion upon normal operation of the videotransmitting apparatus 3, the video receiving apparatus 4, and the videotransmission system 10. On the other hand, if the Isp CPU 411 determinesthat the processing has not been performed on all the regions ofinterest ROI (No), then the Isp CPU 411 goes back to the processing ofstep S41. The Isp CPU 411 repeats the processing from step S41 to stepS49 until the calculating process for the coordinate conversion on allthe regions of interest ROI upon the normal operation is completed.

Next, a correcting process for correcting a defective image upon normaloperation of the video transmitting apparatus 3, the video receivingapparatus 4, and the video transmission system 10 according to thepresent embodiment will be described below with reference to FIG. 18.FIG. 18 is a flowchart illustrating an example of a sequence of acorrecting process for correcting a defective image upon normaloperation of the transmitting apparatus, the receiving apparatus, andthe transmission system. The flowchart illustrated in FIG. 18 representsan example of a sequence of a correcting process for correcting adefective image in one region of interest. In a case where defectivepixels are present in a plurality of regions of interest, the videotransmitting apparatus 3, the video receiving apparatus 4, and the videotransmission system 10 perform the correcting process for correcting adefective image as illustrated in FIG. 18 with respect to each of theregions of interest.

(Step S51)

As illustrated in FIG. 18, in the correcting process for correcting adefective image, first, the static defect correcting section 421acquires the region number (indicated as “ROI ID” in FIG. 18) of theregion of interest ROI input from the Isp CPU 411, and then goes to theprocessing of step S53.

(Step S53)

The static defect correcting section 421 acquires information of defectcoordinates (i.e., information of corrective coordinates) input from theIsp CPU 411 in association with the region number of the region ofinterest ROI, and goes to the processing of step S55.

(Step S55)

The static defect correcting section 421 determines whether thecoordinates of a present image agree with the coordinates of thedefective pixel (i.e., corrective coordinates) acquired in step S53 ornot. Here, the coordinates of the present image represent thecoordinates of an image that the static defect correcting section 421has acquired from the Isp CPU 411. Furthermore, in a case where aplurality of sets of corrective coordinates has been acquired in stepS53, the static defect correcting section 421 determines whether thecoordinates of the present image agree with each of the sets ofcorrective coordinates or not. If the static defect correcting section421 determines that the coordinates of the present image agree with thecorrective coordinates (Yes), then the static defect correcting section421 goes to the processing of step S57. On the other hand, if the staticdefect correcting section 421 determines that the coordinates of thepresent image do not agree with the corrective coordinates (No), thenthe static defect correcting section 421 goes to the processing of stepS59.

(Step S57)

The static defect correcting section 421 corrects the defective imageand then goes to the processing of step S59. In step S57, the staticdefect correcting section 421 corrects the image data where a defect hasoccurred on the basis of the image data of images corresponding topixels at upper, lower, left, and right coordinates adjacent to thedefect coordinates. Specifically, the static defect correcting section421 calculates coordinates of peripheral images around the image at thecoordinates corresponding to the corrective coordinates (an example ofthe defect coordinates) determined as agreeing with the coordinates ofthe present image in step S55. The static defect correcting section 421calculates, as the coordinates of peripheral images, coordinates of apixel that is two rows, for example, ahead of the corrective coordinates(upper coordinates adjacent to the defect coordinates) and coordinatesof a pixel that is two rows, for example, behind of the correctivecoordinates (lower coordinates adjacent to the defect coordinates).Furthermore, the static defect correcting section 421 calculates, as thecoordinates of peripheral images, coordinates of a pixel that is twocolumns, for example, ahead of the corrective coordinates (leftcoordinates adjacent to the defect coordinates) and coordinates of apixel that is two columns, for example, behind of the correctivecoordinates (right coordinates adjacent to the defect coordinates). Whenthe static defect correcting section 421 acquires the image data of theimages at the calculated four sets of coordinates, the static defectcorrecting section 421 corrects the image data of the defective imageusing the acquired image data.

(Step S59)

The static defect correcting section 421 determines whether all thepixels in the region of interest ROI having the region number input instep S51 have been determined as agreeing or disagreeing with thecoordinates of the defective pixel or not. If the static defectcorrecting section 421 determines that all the pixels in the region ofinterest ROI having the region number input in step S51 have beendetermined as agreeing or disagreeing with the coordinates of thedefective pixel (Yes), then the static defect correcting section 421ends the correcting process for correcting a defective image. On theother hand, if the static defect correcting section 421 determines thatall the pixels in the region of interest ROI having the region numberinput in step S51 have not been determined as agreeing or disagreeingwith the coordinates of the defective pixel (No), then the static defectcorrecting section 421 goes to the processing of step S55.

Next, the timing of the correcting process for correcting a defectiveimage in the transmitting apparatus, the receiving apparatus, and thetransmission system according to the present embodiment will bedescribed below with reference to FIG. 19. FIG. 19 is a diagramillustrating an example of a timing chart of the correcting process forcorrecting a defective image in the transmitting apparatus, thereceiving apparatus, and the transmission system according to thepresent embodiment. “SENSOR V Sync” indicated in FIG. 19 represents avertical synchronizing signal input to the sensor CPU 321. “SENSORPROCESSING” indicated in FIG. 19 represents a process performed by thesensor CPU 321. “ISP Sync” indicated in FIG. 19 represents a verticalsynchronizing signal input to the Isp CPU 411. “ISP PROCESSING”indicated in FIG. 19 represents a process performed by the videoreceiving apparatus 4. Regions of interest ROI-ε1 through ROI-ε3indicated in FIG. 19 schematically illustrate regions of interestprocessed in one frame period. In FIG. 19, for an easier understanding,the region of interest ROI-ε1 is processed in the first frame period.For comparison of the sizes of regions of interest, the regions ofinterest ROI-ε2 and ROI-ε3 processed in the second and third frameperiods are also illustrated. Furthermore, FIG. 19 illustrates time asit elapses from the left toward the right.

As illustrated in FIG. 19, the video transmission system 10 has thevideo receiving apparatus 4 request the acquisition of information ofthe coordinates of all defective pixels in the image capturing region(i.e., the full angle of view) provided in the photoelectric convertingsection 311 of the image capturing section 31 in an initializing process(see step S11 illustrated in FIG. 16). Furthermore, the videotransmission system 10 has the video transmitting apparatus 3 acquiredefect coordinates in the image capturing region provided in thephotoelectric converting section 311 (i.e., defect coordinates in thecoordinate system of the image capturing region) in an initializingprocess. Moreover, the video transmission system 10 transmitstransmission data including the information of the defect coordinatesacquired by the video transmitting apparatus 3 to the video receivingapparatus 4, and stores the information of the defect coordinates in thevideo receiving apparatus 4.

As illustrated in FIG. 19, when the sensor CPU 321 detects a framestarting trigger at time t1, the sensor CPU 321 carries out theprocessing of step S31 and step S33 described above as an embeddedsetting process. Specifically, in the embedded setting process, thesensor CPU 321 sets a segmenting position for segmenting an image fromthe image capturing region of the image capturing section 31 andacquires positional information (the coordinates of a left upper end,the length in the X-axis direction, and the length in the Y-axisdirection) in the region of interest ROI-ε1.

The sensor CPU 321 transmits transmission data including embedded datahaving the information set in the embedded setting process to the videoreceiving apparatus 4 by way of communication through hardware (HW)using MIPI at time t2 when the embedded setting process is finished.

When the sensor CPU 321 has started transmitting the transmission data,the sensor CPU 321 starts exposure and reading in the frame, i.e., theimage capturing section 31 starts capturing an image.

At time t3 when the embedded data acquiring section 413 has finisheddecoding the embedded data included in the transmission data that theIsp CPU 411 has started receiving at time t2, the Isp CPU 411 startsacquiring the coordinates of a reference pixel (coordinates of a leftupper end) of the region of interest ROI-ε1 and the size of the regionof interest ROI-ε1 on the basis of the various pieces of informationacquired and input by the embedded data acquiring section 413, andfinishes calculating the coordinates of a corrective pixel present inthe region of interest ROI-ε1 at time t4. In other words, the processingfrom step S41 to step S47 illustrated in FIG. 17 is carried out on oneframe during the period from time t3 to time t4.

The video receiving apparatus 4 carries out the correcting process forcorrecting a defective image and the image quality adjustment accordingto the ISP process from time t4.

Though not described in detail, the correcting process for correcting adefective image is performed on the regions of interest ROI-ε2 andROI-ε3 that are different in position and size at the same timing as theregion of interest ROI-ε1.

As described with reference to FIGS. 16 through 19, the videotransmission system 10 can transmit the transmission data having theembedded data including the defect correcting information regarding theregions of interest ROI from the video transmitting apparatus 3 to thevideo receiving apparatus 4 by way of communication using MIPI. In sucha manner, the video transmission system 10 can perform a demosaicingprocess per region of interest ROI.

5. A Modification of the First Embodiment of the Present Disclosure

Next, transmitting apparatus, receiving apparatus, and transmissionsystems according to modifications of the present embodiment will bedescribed below with reference to FIGS. 20 through 22.

(Modification 1)

A transmitting apparatus, a receiving apparatus, and a transmissionsystem according to Modification 1 of the present embodiment will bedescribed below with reference to FIG. 20. FIG. 20 is a block diagramillustrating a general makeup of a video transmitting apparatus 3, avideo receiving apparatus 4 x, and a transmission system 10 x accordingto the present modification. Incidentally, those components that areidentical in operation and function to those of the video transmittingapparatus 3, the video receiving apparatus 4, and the video transmissionsystem 10 according to the present embodiment are denoted by identicalreference characters, and will be omitted from description.

As illustrated in FIG. 20, the video transmission system 10 x accordingto the present modification includes the video transmitting apparatus 3that is identical in configuration to the video transmitting apparatus 3according to the present embodiment, and the video receiving apparatus 4x that is partly different in configuration from the video receivingapparatus 4 according to the present embodiment. The video receivingapparatus 4 x according to the present modification is characterized inthat it includes a coordinate converting section 43 for convertingdefect coordinates in the image capturing region provided in thephotoelectric converting section 311 into corrective coordinates thatare defect coordinates in the region of interest ROI.

In the video receiving apparatus 4 according to the present embodiment,the coordinate converting section 411 a is provided as a functionalblock of the Isp CPU 411. In contrast, the coordinate converting section43 provided in the video receiving apparatus 4 x ishardware-implemented.

The coordinate converting section 43 is connected to the Isp CPU 411 xand the static defect correcting section 421. Therefore, the coordinateconverting section 43 can acquire the coordinates of a defective pixelin the image capturing region provided in the photoelectric convertingsection 311 and the positional information of the region of interest ROIfrom the Isp CPU 411. In such a manner, the coordinate convertingsection 43 can convert the defect coordinates in the coordinate systemof the image capturing region into defect coordinates (correctivecoordinates) in the coordinate system of the region of interest ROI.Furthermore, the coordinate converting section 43 can output theconverted corrective coordinates to the static defect correcting section421. The coordinate converting section 43 can thus perform the samefunction as the coordinate converting section 411 a according to thepresent embodiment except that the coordinate converting section 43 ishardware-implemented.

Accordingly, the video transmitting apparatus 3 x, the video receivingapparatus 4 x, and the transmission system 10 x according to the presentmodification can correct a defective image present in the region ofinterest ROI in a similar manner to the video transmitting apparatus 3,the video receiving apparatus 4, and the transmission system 10according to the present embodiment though the coordinate convertingsection 43 is hardware-implemented.

(Modification 2)

A transmitting apparatus, a receiving apparatus, and a transmissionsystem according to Modification 2 of the present embodiment will bedescribed below with reference to FIG. 21. FIG. 21 is a block diagramillustrating a general makeup of a video transmitting apparatus 3 y, avideo receiving apparatus 4 y, and a transmission system 10 y accordingto the present modification. Incidentally, those components that areidentical in operation and function to those of the transmittingapparatus, the video receiving apparatus, and the video transmissionsystems according to the present embodiment and Modification 1 aredenoted by identical reference characters, and will be omitted fromdescription.

As illustrated in FIG. 21, the video transmission system 10 y accordingto the present modification is different from the video transmissionsystem 10 according to the present embodiment in that a coordinateconverting section is provided in the video transmitting apparatus 3 y.

The video transmitting apparatus 3 y has a coordinate converting section321 c provided in a sensor CPU 321 y of a controlling section 32 y. Thecoordinate converting section 321 c is connected to a conversion areacontrolling section 321 by. The sensor CPU 321 y according to thepresent modification is configured to output the positional informationof a region of interest ROI as defect correcting information from theconversion area controlling section 321 by to the coordinate convertingsection 321 c upon normal operation of the video transmitting apparatus3 y.

The conversion area controlling section 321 by is configured to acquireand hold, as the defect correcting information, defect coordinatesrepresenting the coordinates of a pixel where a defect has occurredamong a plurality of pixels that make up the image capturing regionprovided in the photoelectric converting section 311. The coordinateconverting section 321 c converts the coordinates of the defective pixel(defect coordinates) in the image capturing region provided in thephotoelectric converting section 311 into coordinates of the defectivepixel (corrective coordinates) in the region of interest ROI, using thepositional information of the region of interest ROI input from theconversion area controlling section 321 by and the coordinates of thedefective pixels read from the nonvolatile storage device 33. Theconversion area controlling section 321 by and the coordinate convertingsection 321 c that have such functions correspond to an example of thecontrolling section. Furthermore, the sensor CPU 321 that has theconversion area controlling section 321 by and the coordinate convertingsection 321 c and the controlling section 32 y that has the sensor CPU321 correspond to an example of the controlling section.

The coordinate converting section 321 c determines a position in theregion of interest ROI that corresponds to the coordinates of a pixelwhere a defect has occurred (according to the present embodiment, thecoordinates in the XY orthogonal coordinate system of the region ofinterest ROI) on the basis of the reference coordinates of the region ofinterest ROI (for example, the coordinates of the left upper end and thesize (for example, the length in the X-axis direction, and the length inthe Y-axis direction) thereof and the coordinates of the pixel where thedefect has occurred (according to the present embodiment, thecoordinates in the XY orthogonal coordinate system of the imagecapturing region). Since the coordinate converting section 321 cconverts the defect coordinates into the corrective coordinates in asimilar manner to the coordinate converting section 411 a according tothe present embodiment, the description thereof will be omitted. Thecoordinate converting section 321 c outputs the corrective coordinatesobtained by the coordinate conversion to the transmitting section 322.

The transmitting section 322 includes ROI information that includes thecorrective coordinates at the coordinates in the region of interest ROIand the positional information of the region of interest ROI, input fromthe coordinate converting section 321 c, in the embedded data inassociation with the region number of the region of interest ROI, andsend out transmission data having the embedded data to the videoreceiving apparatus 4 y. In other words, the transmitting section 322includes information of the position of the pixel where the defect hasoccurred in the region of interest ROI determined by the coordinateconverting section 321 c (i.e., the corrective coordinates at thecoordinates in the region of interest ROI) in the ROI information, andsends out the ROI information.

The sensor CPU 321 outputs the defect correcting information includingthe information of the defect coordinates (e.g., the correctivecoordinates) to the transmitting section 322. Furthermore, the sensorCPU 321 outputs the information of the basic coordinates and size of theregion of interest ROI (i.e., the positional information of the regionof interest ROI) as the defect correcting information to thetransmitting section 322. The transmitting section 322 generatestransmission information (see FIGS. 6 and 12) including the defectcorrecting information and the information of the region of interest ROIinput from the sensor CPU 321, and sends out the generated transmissioninformation to the video receiving apparatus 4. The defect correctinginformation is included in the ROI information and sent out from thetransmitting section 322. Since the ROI information is included in theembedded data, the defect correcting information is included in theembedded data and sent out from the transmitting section 322. In thisfashion, the video transmitting apparatus 3 sends out the defectcorrecting information included in the ROI information to from thetransmitting section 322.

An embedded data acquiring section (an example of a controlling section)413 y provided in the video receiving apparatus 4 y is configured toacquire the coordinates of a pixel where a defect has occurred among aplurality of pixels that make up the image capturing region provided inthe photoelectric converting section 311 as defect correctinginformation. Specifically, the embedded data acquiring section 413 yacquires the coordinates (corrective coordinates) of a pixel (correctivepixel) corresponding to an image as a correction target from the ROIinformation included in the embedded data that the transmission datainput via the receiving section 412 has, as defect correctinginformation, and outputs the acquired defect correcting information tothe static defect correcting section 421. The coordinates of the pixelincluded in the ROI information are coordinates in the XY orthogonalcoordinate system of the region of interest ROI. Therefore, the embeddeddata acquiring section 413 y extracts and acquires the coordinates ofthe pixel where the defect has occurred as coordinates in the region ofinterest ROI. The embedded data acquiring section 413 y outputscorrective coordinates corresponding to the region number (indicated as“ROI ID” in FIG. 21) of the region of interest ROI per region number tothe static defect correcting section 421. Inasmuch as the imageprocessing section 42 according to the present modification is of asimilar configuration and is configured to perform a similar function tothe image processing section 42 including the static defect correctingsection 421 according to the present embodiment, the description thereofwill be omitted.

In such a manner, with the video transmitting apparatus 3 y, the videoreceiving apparatus 4 y, and the transmission system 10 y, thecoordinate converting section 321 c is provided in the videotransmitting apparatus 3 y. The video transmitting apparatus 3 y, thevideo receiving apparatus 4 y, and the transmission system 10 y cancorrect a defective image present in the region of interest ROI in asimilar manner to the video transmitting apparatus 3, the videoreceiving apparatus 4, and the video transmission system 10 according tothe present embodiment.

(Modification 3)

A transmitting apparatus, a receiving apparatus, and a transmissionsystem according to Modification 3 of the present embodiment will bedescribed below with reference to FIG. 22. FIG. 22 is a block diagramillustrating a general makeup of a video transmitting apparatus 3 z, avideo receiving apparatus 4 z, and a video transmission system 10 zaccording to the present modification. Incidentally, those componentsthat are identical in operation and function to those of thetransmitting apparatus, the video receiving apparatus, and the videotransmission systems according to the present embodiment, Modification1, or Modification 2 are denoted by identical reference characters, andwill be omitted from description.

The video transmitting apparatus 3 z according to the presentmodification is characterized in that it has a static defect correctingsection (an example of a controlling section) 34 that performs a similarfunction to the static defect correcting section 421 according to thepresent embodiment and a Raw processing section 35 that performs part ofthe function of the image generating section 422 according to thepresent embodiment. The video receiving apparatus 4 z according to thepresent modification is characterized in that it does not have anembedded data acquiring section and a static defect correcting sectionand has an RGB processing section 424 that performs part of the functionof the image generating section 422 according to the present embodiment.

The static defect correcting section 34 included in the videotransmitting apparatus 3 z is connected to the amplifying section 313provided in the image capturing section 31, the conversion areacontrolling section 321 b and the coordinate converting section 321 cprovided in the sensor CPU 321 y, and the Raw processing section 35. TheRaw processing section 35 is connected to the transmitting section 322provided in the controlling section 32 y.

The static defect correcting section 34 performs a correcting processfor correcting a defective image corresponding to a defective pixel,using the positional information of the region of interest ROI inputfrom the conversion area controlling section 321 b, the coordinates ofthe defective pixel (corrective coordinates) in the region of interestROI input from the coordinate converting section 321 c, and the imagedata input from the amplifying section 313. The static defect correctingsection 34 is configured to perform a correction of the image data of animage where a defect has occurred on the basis of the image data ofimages corresponding to pixels at upper, lower, left, and rightcoordinates adjacent to the position of the pixel where the defect hasoccurred in the region of interest ROI determined by the coordinateconverting section 321 c (an example of the controlling section). Inother words, the static defect correcting section 34 corrects adefective image in a similar manner to the static defect correctingsection 421 according to the present embodiment. Therefore, the staticdefect correcting section 34 will be omitted from description. Thetransmitting section 322 includes the image data of the image correctedby the static defect correcting section 34 in the ROI information, andsends out the ROI information.

When the image data of the region of interest ROI including the imagedata where the defect has been corrected are input, the Raw processingsection 35 performs a Raw process to generate a region of interest ROIrepresented by a Raw image. The Raw processing section 35 outputs theimage data of the generated Raw image to the transmitting section 322.

The transmitting section 322 is configured to generate transmission datahaving embedded data including the positional information of the regionof interest ROI input from the sensor CPU 321 y and associated with theregion number of the region of interest ROI and payload data includingthe image data of the Raw image input from the Raw processing section 35and associated with the region number, and send out the generatedtransmission data to the video receiving apparatus 4 y.

The receiving section 412 provided in the video receiving apparatus 4 zextracts the embedded data and the payload data from the transmissiondata input from the transmitting section 322 and outputs the embeddeddata and the payload data to an Isp CPU 411 z. The Isp CPU 411 zextracts the positional information of the region of interest ROI fromthe input embedded data, and extracts the image data of the Raw imagefrom the input payload data. The Isp CPU 411 z outputs the extractedregion number and positional information of the region of interest ROIand the extracted image data of the Raw image to the RGB processingsection 424 provided in an image processing section 42 z.

The RGB processing section 424 performs an RGB process using theinformation and image data input from the Isp CPU 411 z to generate animage of the region of interest ROI including image data of RGB signals.The RGB processing section 424 outputs the generated image data to theimage quality adjusting section 423. The image quality adjusting section423 is of a similar configuration and is configured to perform a similarfunction to the image quality adjusting section 423 according to thepresent embodiment, and hence the description thereof will be omitted.

With the video transmitting apparatus 3 z, the video receiving apparatus4 z, and the video transmission system 10 z, the static defectcorrecting section 34 and the Raw processing section 35 are provided inthe video transmitting apparatus 3 z. However, the video transmittingapparatus 3 z, the video receiving apparatus 4 z, and the videotransmission system 10 z can correct a defective image present in theregion of interest ROI in a similar manner to the video transmittingapparatus 3, the video receiving apparatus 4, and the video transmissionsystem 10 according to the present embodiment.

As described above, the transmitting apparatus, receiving apparatus, andtransmission systems according to the present embodiment and themodifications can realize a correcting process for correcting adefective image in a region of interest that is a partial regionsegmented from a captured image.

6. Second Embodiment of the Present Disclosure

Next, a transmitting apparatus, a receiving apparatus, and atransmission system according to a second embodiment of the presentdisclosure will be described below with reference to FIG. 23. With thetransmitting apparatus, the receiving apparatus, and the transmissionsystem according to the first embodiment, an object as a target to besegmented from an image capturing region is of a rectangular shape. Incontrast, the transmitting apparatus, the receiving apparatus, and thetransmission system according to the present embodiment arecharacterized in that they perform a correcting process for correcting adefective image in a case where an object as a target to be segmentedfrom an image capturing region is not of a rectangular shape.

FIG. 23 is a diagram schematically illustrating a method of calculatingthe position of a defective pixel in a region of interest on the basisof coordinate information of the defective pixel acquired in a deliveryinspection or the like according to the present embodiment. In FIG. 23,for an easier understanding, only pixels established in a region ofinterest ROI among pixels arrayed in a matrix in an image capturingregion IR in its entirety are illustrated. Furthermore, in FIG. 23, foran easier understanding, a defective pixel D11 is illustrated in across-hatched manner, and a segmentation target CO as an object to besegmented from the image capturing region IR is illustrated in astippled manner.

As illustrated in FIG. 23, the segmentation target CO as an object to besegmented from the image capturing region IR is not of a rectangularshape. One of the pixels included in the segmentation target CO is thedefective pixel D11. According to the present embodiment, in a casewhere the segmentation target CO is not of a rectangular shape, theregion of interest ROI is established as being of a minimum rectangularshape including the segmentation target CO, as described above in thepresupposed technology 2. As described above in the first embodiment, ina case where the shape of a region of interest and the shape of asegmentation target are in agreement with each other, defect coordinatesin the image capturing region are converted into defect coordinates inthe region of interest frame by frame. In contrast, in a case where theshape of a region of interest and the shape of a segmentation target arenot in agreement with each other, defect coordinates in the imagecapturing region are converted into defect coordinates in the region ofinterest row by row.

As illustrated in FIG. 23, the image capturing region IR has its originIRo (0, 0) located on a pixel at a left upper end thereof. In this case,the defective pixel D11 in the image capturing region IR has coordinates(x, y) represented respectively by (D1_X1, D1_Y1). The coordinates(D1_X1, D1_Y1) of the defective pixel D11 are coordinates based on theorigin IRo (0, 0) of the image capturing region IR. Furthermore, thecoordinates (D1_X1, D1_Y1) of the defective pixel D11 have informationas coordinate information acquired upon a delivery inspection, forexample, of the video transmitting apparatus that has the imagecapturing section.

In a case where a region of interest is established as described above,positional information of the region of interest (for example, thecoordinates of a left upper end thereof, the length in the X-axisdirection thereof, and the length in the Y-axis direction thereof) isderived. Consequently, as illustrated in FIG. 23, the coordinates (R1_X,R1Y) of a left upper end Pr, the length R1_W in the X-axis direction,and the length R1_H in the Y-axis direction, for example, are derived aspositional information of the region of interest ROI1. The coordinates(R1_X, R1Y) of the left upper end Pr of the region of interest ROI1represent the position of the region of interest ROI1 in the imagecapturing region IR. Therefore, the coordinates (R1_X, R1Y) of the leftupper end Pr are coordinates based on the origin IRo (0, 0) of the imagecapturing region IR.

According to the present embodiment, in addition to establishingcoordinates of the defective pixel and the region of interest in theimage capturing region, coordinates of a pixel at an end (which mayhereinafter be referred to as “end pixel”) of the segmentation target inthe region of interest are also established. According to the presentembodiment, an end pixel is established with respect to a left end ofthe region of interest, for example. As illustrated in FIG. 23, thecoordinates of an end pixel Pe are established as coordinates (D_XR, 0).The X coordinate of the end pixel Pe corresponds to the number of pixelsfrom the left end of the region of interest ROI to the end pixel.Therefore, depending on the shape of the segmentation target CO, thecoordinates of the end pixel Pe may be different from row to row.

The coordinates (D1X_R1, D1_YR1) of the defective pixel D11 in theregion of interest ROI can be expressed using the coordinates (R1_X,R1Y) of the left upper end Pr of the region of interest ROI1, thecoordinates (D1_X1, D1_Y1) of the defective pixel D11, and thecoordinates (D_XR, 0) of the end pixel Pe by the following equations (3)and (4):

D1_XR1=D1_X−R1_X−D_XR  (3)

D1_YR1=D1_Y−R1_Y−0  (4)

Because of the equations (3) and (4), the position of the defectivepixel D11 can be expressed by coordinates in the coordinate space of theregion of interest ROI1. Therefore, by storing coordinate informationacquired upon a delivery inspection or the like, for example, the videoreceiving apparatus can calculate coordinates of a defective pixel in aregion of interest each time positional information (the coordinates ofa left upper end, the length in the X-axis direction, and the length inthe Y-axis direction) of the established region of interest and thepositions of the pixels at ends of the rows included in the region ofinterest are input from the video transmitting apparatus.

The transmitting apparatus according to the present embodiment mayinclude positional information (e.g., the coordinates of the end pixelPe) of a target object as in the presupposed technology 2, for example,in the payload per pixel row and send out the payload data to the videoreceiving apparatus. Furthermore, since an object as a segmentationtarget is not of a rectangular shape, there may be an instance whereupper, lower, left, and right images may not be present around an imagecorresponding to a pixel (corrective pixel) that corresponds to an imageas a correction target. In this case, the correcting process forcorrecting a defective image may be carried out by interpolating imagedata according to a boundary process, for example. For example, anonexistent one of upper and lower images adjacent to a corrective imageis interpolated with the existent image, whereas a nonexistent one ofleft and right images adjacent to the corrective image is interpolatedwith the existent image.

The configurations of either one of the transmitting apparatus, thereceiving apparatus, and the transmission systems according to the firstembodiment and Modifications 1 through 3 may be applied to thetransmitting apparatus, the receiving apparatus, and the transmissionsystem according to the present embodiment. Therefore, theconfigurations of the transmitting apparatus, the receiving apparatus,and the transmission system according to the present embodiment will beomitted from description.

As described above, the transmitting apparatus, the receiving apparatus,and the transmission system according to the present embodiment canrealize a correcting process for correcting a defective image in aregion of interest that is a partial region segmented from a capturedimage even if an object as a segmentation target is not of a rectangularshape.

The present disclosure has been described above with respect to thepresupposed technologies, the embodiments, and the modification.However, the present disclosure is not limited to the above embodimentsetc., but various changes and modifications may be made therein. It isnoted that the advantages set forth in the present description are givenby way of illustrative example only. The advantages of the presentdisclosure are not limited to those set forth in the presentdescription. The present disclosure may have other advantages than theadvantages set forth in the present description.

Furthermore, the present disclosure may have the following arrangements,for example:

(1)

A transmitting apparatus including:

a controlling section that controls holding of defect correctinginformation for use in correcting a defect in an image included in a ROI(Region Of Interest); and

a transmitting section that sends out image data of the image includedin the ROI as payload data and sends out ROI information as embeddeddata.

(2)

The transmitting apparatus according to (1), in which the defectcorrecting information is included in the ROI information and sent outfrom the transmitting section.

(3)

The transmitting apparatus according to (1) or (2), in which thecontrolling section holds coordinates of a pixel where a defect hasoccurred among a plurality of pixels making up an image capturing regionas the defect correcting information.

(4)

The transmitting apparatus according to (3), in which the controllingsection holds the coordinates of the pixel where the defect has occurredwhen the transmitting apparatus is initialized or manufactured.

(5)

The transmitting apparatus according to (3) or (4), in which thecontrolling section holds reference coordinates and size of the ROI asthe defect correcting information.

(6)

The transmitting apparatus according to (5), in which the controllingsection holds left upper end coordinates of the ROI as the referencecoordinates.

(7)

The transmitting apparatus according to (5) or (6), in which thecontrolling section determines a position in the ROI that corresponds tothe coordinates of the pixel where the defect has occurred on the basisof the reference coordinates and the size of the ROI and the coordinatesof the pixel where the defect has occurred.

(8)

The transmitting apparatus according to (7), in which the transmittingsection includes information of the position of the pixel where thedefect has occurred in the ROI determined by the controlling section andthe image data of the image included in the ROI in the ROI information,and sends out the ROI information.

(9)

The transmitting apparatus according to (7), further including:

a processing section that processes a correction of the image data ofthe image where the defect has occurred on the basis of image data ofimages corresponding to pixels at upper, lower, left, and rightcoordinates adjacent to the coordinates corresponding to the position ofthe pixel where the defect has occurred in the ROI determined by thecontrolling section,

in which the transmitting section includes the image data of the imagecorrected by the processing section in the ROI information, and sendsout the ROI information.

(10)

The transmitting apparatus according to any one of (1) through (9), inwhich the transmitting section sends out a signal according to MIPI(Mobile Industry Processor Interface) D-PHY standards, MIPI C-PHYstandards, or MIPI CSI (Camera Serial Interface)-2 standards.

(11)

A receiving apparatus including:

a receiving section that receives a transmission signal including imagedata of an image included in a ROI (Region Of Interest) in payload dataand including ROI information in embedded data;

a controlling section that controls extraction of defect correctinginformation for use in correcting a defect in the image data of theimage included in the ROI from the transmission signal received by thereceiving section; and

a processing section that processes a correction of the defect in theimage of the ROI on the basis of the defect correcting informationextracted by the controlling section.

(12)

The receiving apparatus according to (11), in which the controllingsection extracts the defect correcting information from the ROIinformation included in the transmission signal.

(13)

The receiving apparatus according to (11), in which the controllingsection extracts, as the defect correcting information, coordinates of apixel where a defect has occurred among a plurality of pixels making upan image capturing region.

(14)

The receiving apparatus according to (13), in which the controllingsection extracts the coordinates of the pixel where the defect hasoccurred as coordinates in either the image capturing region or the ROI.

(15)

The receiving apparatus according to any one of (11) through (14), inwhich the controlling section extracts reference coordinates and size ofthe ROI as the defect correcting information.

(16)

The receiving apparatus according to (15), in which the controllingsection extracts left upper end coordinates of the ROI as the referencecoordinates.

(17)

The receiving apparatus according to (15) or (16), in which thecontrolling section determines a position in the ROI that corresponds tothe coordinates of the pixel where the defect has occurred on the basisof the reference coordinates and the size of the ROI and the coordinatesof the pixel where the defect has occurred.

(18)

The receiving apparatus according to any one of (13) through (17), inwhich the processing section corrects the image data of the image wherethe defect has occurred on the basis of the image data of imagescorresponding to pixels at upper, lower, left, and right coordinatesadjacent to the coordinates of the pixel where the defect has occurred.

(19)

The receiving apparatus according to any one of (11) through (18), inwhich the receiving section receives a signal according to MIPI (MobileIndustry Processor Interface) D-PHY standards, MIPI C-PHY standards, orMIPI CSI (Camera Serial Interface)-2 standards.

(20)

A transmission system including:

a transmitting apparatus having a controlling section that controlsholding of defect correcting information for use in correcting a defectin an image included in a ROI (Region Of Interest), and a transmittingsection that sends out image data of the image included in the ROI aspayload data and sends out ROI information as embedded data; and

a receiving apparatus having a receiving section that receivestransmission signal including the image data of the image included inthe ROI in the payload data and including the ROI information in theembedded data, a controlling section that controls extraction of defectcorrecting information for use in correcting the defect in the imagedata of the image included in the ROI from the transmission signalreceived by the receiving section, and a processing section thatprocesses a correction of the defect in the image of the ROI on thebasis of the defect correcting information extracted by the controllingsection.

It will be understood that those skilled in the art can anticipatevarious corrections, combinations, sub-combinations, and changesdepending on design requirements and other factors as falling within thescope of attached claims and the scope of their equivalents.

REFERENCE SIGNS LIST

-   -   1, 10, 10 x, 10 y, 10 z: Video transmission system    -   3, 3 y, 3 z, 100: Video transmitting apparatus    -   4, 4 x, 4 y, 4 z, 3 z, 200: Video receiving apparatus    -   31, 110: Image capturing section    -   32, 32 y, 41, 41 z: Controlling section    -   34, 421: Static defect correcting section    -   35: Raw processing section    -   42, 42 z: Image processing section    -   43, 321 c, 411 a: Coordinate converting section    -   100A: CSI transmitter    -   100B: CCI slave    -   111: Captured image    -   112, 112 al, 112 a 2, 112 a 3, 112 a 4, 112 b 1, 112 b 4, 123 a        4, 223A: ROI image    -   112 b: Compressed image data    -   113, 114: Positional information    -   115: Priority    -   116, 116 a 1, 116 a 2: Transmission image    -   118: Image    -   120, 130: Image processing section    -   120A, 120A1, 120A2, 130A, 147B: Compressed image data    -   120B: ROI information    -   120C: Frame information    -   121: ROI segmenting section    -   122: ROI analyzing section    -   123: Detecting section    -   124: Priority setting section    -   125, 131: Encoding section    -   126: Image processing controlling section    -   140: Transmitting section    -   141: LINK controlling section    -   142: ECC generating section    -   143: PH generating section    -   144, 145: ROI data buffer    -   144: EBD buffer    -   146: Normal image data buffer    -   147: Combining section    -   147A: Transmission data    -   200A: CSI receiver    -   200B: CCI master    -   210: Receiving section    -   211: Header separating section    -   212: Header interpreting section    -   213: Payload separating section    -   214: EBD interpreting section    -   214A: EBD data    -   215: ROI data separating section    -   215A, 215B: Payload data    -   220: Information processing section    -   221: Information extracting section    -   221A: Extracted information    -   222: ROI decoding section    -   222A: Image data    -   223: ROI image generating section    -   224: Normal image decoding section    -   224A: Normal image    -   311: Photoelectric converting section    -   312: Signal converting section    -   313: Amplifying section    -   321, 321 y: Sensor CPU    -   321 a: Exposure controlling section    -   321 b, 321 by: Conversion area controlling section    -   322: Transmitting section    -   411, 411 x, 411 y: Isp CPU    -   412: Receiving section    -   413, 413 y: Embedded data acquiring section    -   422: Image generating section    -   423: Image quality adjusting section    -   423 a: Coordinate determining section    -   424: RGB processing section    -   ADC: Analog-to-digital conversion    -   AGC: Signal amplification    -   CCI: Camera control interface    -   CL: Clock lane

1. A transmitting apparatus comprising: a controlling section thatcontrols holding of defect correcting information for use in correctinga defect in an image included in a ROI (Region Of Interest); and atransmitting section that sends out image data of the image included inthe ROI as payload data and sends out ROI information as embedded data.2. The transmitting apparatus according to claim 1, wherein the defectcorrecting information is included in the ROI information and sent outfrom the transmitting section.
 3. The transmitting apparatus accordingto claim 1, wherein the controlling section holds coordinates of a pixelwhere a defect has occurred among a plurality of pixels making up animage capturing region as the defect correcting information.
 4. Thetransmitting apparatus according to claim 3, wherein the controllingsection holds the coordinates of the pixel where the defect has occurredwhen the transmitting apparatus is initialized or manufactured.
 5. Thetransmitting apparatus according to claim 3, wherein the controllingsection holds reference coordinates and size of the ROI as the defectcorrecting information.
 6. The transmitting apparatus according to claim5, wherein the controlling section holds left upper end coordinates ofthe ROI as the reference coordinates.
 7. The transmitting apparatusaccording to claim 5, wherein the controlling section determines aposition in the ROI that corresponds to the coordinates of the pixelwhere the defect has occurred on a basis of the reference coordinatesand the size of the ROI and the coordinates of the pixel where thedefect has occurred.
 8. The transmitting apparatus according to claim 7,wherein the transmitting section includes information of the position ofthe pixel where the defect has occurred in the ROI determined by thecontrolling section and the image data of the image included in the ROIin the ROI information, and sends out the ROI information.
 9. Thetransmitting apparatus according to claim 7, further comprising: aprocessing section that processes a correction of the image data of theimage where the defect has occurred on a basis of image data of imagescorresponding to pixels at upper, lower, left, and right coordinatesadjacent to the coordinates corresponding to the position of the pixelwhere the defect has occurred in the ROI determined by the controllingsection, wherein the transmitting section includes the image data of theimage corrected by the processing section in the ROI information, andsends out the ROI information.
 10. The transmitting apparatus accordingto claim 1, wherein the transmitting section sends out a signalaccording to MIPI (Mobile Industry Processor Interface) D-PHY standards,MIPI C-PHY standards, or MIPI CSI (Camera Serial Interface)-2 standards.11. A receiving apparatus comprising: a receiving section that receivesa transmission signal including image data of an image included in a ROI(Region Of Interest) in payload data and including ROI information inembedded data; a controlling section that controls extraction of defectcorrecting information for use in correcting a defect in the image dataof the image included in the ROI from the transmission signal receivedby the receiving section; and a processing section that processes acorrection of the defect in the image of the ROI on a basis of thedefect correcting information extracted by the controlling section. 12.The receiving apparatus according to claim 11, wherein the controllingsection extracts the defect correcting information from the ROIinformation included in the transmission signal.
 13. The receivingapparatus according to claim 11, wherein the controlling sectionextracts, as the defect correcting information, coordinates of a pixelwhere a defect has occurred among a plurality of pixels making up animage capturing region.
 14. The receiving apparatus according to claim13, wherein the controlling section extracts the coordinates of thepixel where the defect has occurred as coordinates in either the imagecapturing region or the ROI.
 15. The receiving apparatus according toclaim 11, wherein the controlling section extracts reference coordinatesand size of the ROI as the defect correcting information.
 16. Thereceiving apparatus according to claim 15, wherein the controllingsection extracts left upper end coordinates of the ROI as the referencecoordinates.
 17. The receiving apparatus according to claim 15, whereinthe controlling section determines a position in the ROI thatcorresponds to the coordinates of the pixel where the defect hasoccurred on a basis of the reference coordinates and the size of the ROIand the coordinates of the pixel where the defect has occurred.
 18. Thereceiving apparatus according to claim 13, wherein the processingsection corrects the image data of the image where the defect hasoccurred on a basis of the image data of images corresponding to pixelsat upper, lower, left, and right coordinates adjacent to the coordinatesof the pixel where the defect has occurred.
 19. The receiving apparatusaccording to claim 11, wherein the receiving section receives a signalaccording to MIPI (Mobile Industry Processor Interface) D-PHY standards,MIPI C-PHY standards, or MIPI CSI (Camera Serial Interface)-2 standards.20. A transmission system comprising: a transmitting apparatus having acontrolling section that controls holding of defect correctinginformation for use in correcting a defect in an image included in a ROI(Region Of Interest) and a transmitting section that sends out imagedata of the image included in the ROI as payload data and sends out ROIinformation as embedded data; and a receiving apparatus having areceiving section that receives transmission signal including the imagedata of the image included in the ROI in the payload data and includingthe ROI information in the embedded data, a controlling section thatcontrols extraction of defect correcting information for use incorrecting the defect in the image data of the image included in the ROIfrom the transmission signal received by the receiving section, and aprocessing section that processes a correction of the defect in theimage of the ROI on a basis of the defect correcting informationextracted by the controlling section.